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23rd Annual National Test & Evaluation Conference 
“Test & Evaluation in Support of Operational Suitability, Effectiveness and Sustainment of Deployed Systems” 
Hilton Head Island, SC 


12 - 15 March 2007 


Agenda 


MONDAY, MARCH 12, 2007 


TUTORIALS - SESSIONS T1 - T5 


Tutorial T1. Innovative Acquisition Methodologies in Support of Test & Evaluation Objectives - Mr. Jeffrey Manthos, Program Analyst, GSA 
Tutorial T2. The Critical Link Between Test & Evaluation and Modeling & Simulation - Dr. Mark Kiemele, President and Co-Founder, Air Academy 
Associates 

Tutorial T4. DoD Acquisition Policy on Suitability KPP and Related Issues - Dr. Paul, Alfieri, Director of Research, DAU 


TUTORIALS: Sessions T5 - T8 (Continued Sessions) 
Tutorial TS. Innovative Acquisition Methodologies in Support of Test & Evaluation Objectives - Mr. Jeffrey Manthos, Program Analyst, GSA 
Tutorial T6. The Critical Link Between Test & Evaluation and Modeling & Simulation - Dr. Mark Kiemele, President and Co-Founder, Air Academy 


Associates 
Tutorial T8. DoD Acquisition Policy on Suitability KPP and Related Issues - Dr. Paul Alfieri, Director of Research, DAU 


TUESDAY, MARCH 13, 2007 


Session A 


Conference Keynote Address: Assuring Quality and Sustainability in a Competitive Market - Mr. Keith E. Williams, President & CEO, Underwriters 
Laboratories 


Session B 


OTA’s Roundtable: Military Perspectives on Testing for Sustainability and Life Cycle Costing 
Chair: MG James Myles, USA, Commanding General ATEC 

Panelist: RDML William McCarthy, USN, Commander OPTEVFOR 

Mr. Jerry Kitchen, Commander AFOTEC 

COL Michael Bohn, USMC, Commander MCOTEA 


Session C 


Statistically-Based Test Optimization Strategies - Dr. Neal Mackertich, Raytheon Integrated Defense Systems 
Report from the NDIA Industrial Committee on Operational Test & Evaluation (ICOTE) - Mr. Larry Graviss, Vice President, Jacobs Technology, Inc. 


Concurrent Focus Sessions 


Session D 


Planning & Implementing Systems Engineering/T&E to Address Sustainability 


FS D1.1 T&E for Verifying Technology Development and Maturation - Mr. Chris DiPetto, Deputy Director for DT&E, OUSD (AT&L)/SSE 

FS D1.3 Using Technology Readiness Assessments (TRA’s) to Assess the Maturity of Life Cycle Related Technologies - Dr. Jay Mandelbaum, IDA 

FS D1.4 Results of FY 2006 Test Resources Management Center Range Encroachment Survey Provided to the August 2006 TRMC Infrastructure Review - 
Mr. William Egan, IDA 

FS D1.5 Test and Training Enabling Architecture, Use to Prototype Joint Mission Environment Test Capability (JMETC) - Mr. Richard Lockhart, Defense 
Test Resource Management Center 

FS D1.6 TRACER: A Tradespace Analysis Framework - Mr. Gerry Belcher, LMI 


Reducing Total Cost of Ownership & Realistic Estimates of Reliability 


23rd Annual National Test & Evaluation Conference.html[6/14/2016 9:42:55 AM] 


Untitled Document 


e FS D2.1 Low Cost Epoch-by-Epoch Network Centric Positioning Unit (ENPU) for FCS Testing - Dr. Jeffrey Fayman, Geodetics, Inc. and Dr. Lydia Bock, 
Geodetics, Inc. 

e FS D2.2 Testing of the GPS SAASM End-to-End Functionality on Operational Weapon Systems Without the Availability of Signal in Space - Mr. Jim 
Killian, 46 Test Squadron, Holloman AFB 

e FS D2.4 A Successfully Implemented Coordinated Subsystem Reliability Growth Planning Approach- Mr. Louis Chenard, Bombardier & Mr. Paul Ellner, 
AMSAA 

e FS D2.5 Understanding the Combined Influence on Ownership Cost of Reliability, Maintainability, Component Packaging, Commonality, and Process 
Performance - Dr. James A. Forbes, LMI 

e FS D2.6 Using Mission Essential Task Lists as a Basis for Mission-Based Operational Test Planning - Mr. James Carpenter, AVW Technologies 


Test Technology 


FS D3.1 Hazard Assessment of the SM-3 Block IA Missile - Mr. William Houchins, NSWC Dahlgren 

FS D3.2 The Four Element Framework: An Integrated Test & Evaluation Strategy - Mr. Christopher Wilcox, ATEC, APG 

FS D3.3 Operability Testing for High Availability Systems - Mr. Edward Beck, CSC 

FS D3.5 746 Test Squadron Testing of the Latest Embedded GPS/INS (EGI) Hybrid 

Navigation System for the F-16 Fighting Falcon - Mr. Jim Killian, 46 Test Squadron, Holloman AFB 

FS D3.6 Qualification Testing of the EFSS 120mm Mortar Ammunition - Mr. Patrick Freemyers, NSWC Dahlgren 

FS D3.7 Operational Analysis of Modern Condition Based Maintenance (CBM) to Enhance System Sustainability - Dr. Don Gaver, NPS 


WEDNESDAY, MARCH 14, 2007 


Session E 


e Advancing Survivability - Mr. Regis Luther, Armor Holdings 
e A Concept for Suitability Assessment in Capabilities-Based T&E - Dr. Vincent P. Roske, Institute for Defense Analyses 


Session F 


e Development, Test & Evaluation of Unmanned Ground Vehicles/Robots, VADM Joseph Dyer, USN (Ret), President and General Manager, iRobot 
Corporation 
e Industry Panel: 
e Sustainability KPP: Industry Weighs in on Bidding for It and Building for It - Mr. Steve Zink (For Mr. John Stoddart) 
o Past, Present & Future Trends in RM&A - Mr. Ron Mutzelburg, Vice President, Boeing Phantom Works, DC 


Session G 


e Coping with Reliability, Sustainability and Life Cycle Costs at DHS - Dr. George Zarur, Science Advisor, Science and Technology, DHS 
e DT&E Panel: Systems Engineering and DT&E for System Sustainability 
Chair: Col Richard Stuckey, USAF, Principal Assistant Deputy Director for DT&E, OUSD (AT&L)/SSE 


Concurrent Focus Sessions 
Session H 
Test Methodology 


e FS H4.1 Revisiting Quantitative Methods for Evaluating Training Programs for Systems Undergoing OT - Dr. Christopher Hekimian, SAIC 

e FS H4.2 Producing Anywhere, Anytime Test, Evaluation and Diagnostics Capable Products to Eliminate the T&E Logistic Burden - Mr. Ryan Kinney, 
Agilent Technologies 

e FS H4.3 Ship Suitability Testing - Preparing for the Future - Mr. Steve Fisher, NAVAIR, Patuxent River 

e FS H4.4 Innovative Techniques to Solve Measurement and Detection Problems - Dr. Michael Slocum, Air Academy Associates 


Test Planning 


e FS H5.1 A Continuum of Testing for Military Systems - Dr. Patricia Jacobs, NPS 

FS H5.2 CTEC (Collaborative Test & Evaluation Capability) and the Event Streaming Media System - Mr. Reid Johnson, NUWC Keyport 

FS H5.3 Integrated Weapons System Testing - Ms. Laura DeSimone, NSWC Dahlgren 

FS H5.4 Estimating Durations and Trials to Success in Test Trial Programs - Dr. Danny Hughes, LMI 

FS H5.5 Operational Testing of Procedures is Critical to Effective Use/Integration of Emerging Capabilities: Report on Joint Space Control Operations - Col 
Christian Daehnick, USAF, Negation Joint Test & Evaluation 


Technology to Reduce Life Cycle Cost 


e FS H6.1 T&E of Electromagnetic Railguns - Mr. Thomas Boucher, NSWC Dahlgren 

e FS H6.2 Cost Efficient Risk Management Through Integrated T&E Throughout the Systems Engineering Life-Cycle - Mr. Joseph Tribble, AVW 
Technologies 

e FS H6.3 Sustainment of the Deployed Navy Munitions Inventory Through Continuous Quality Evaluation Test & Evaluation - Mr. Jeffrey Johnson, Navy 
Indianhead 


THURSDAY, MARCH 15, 2007 
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Session I 


e Suitability - At What Cost? - Dr. Paul Alfieri, DAU 
Service T&E Executives Speak 
o Mr. David Hamilton, Deputy Director, AF/TE 
o Mr. Steven Whitehead, OPTEVFOR 
e The Costs of Unsuitability and Benefits of Building in Reliability, Availability and Maintainability - Dr. Ernest Seglie, DOT&E 
e Army T&E - Providing Essential Information to an Army at War - Mr. Brian Simmons, Army DTC 


Session J 


e Reliability-Based Design, Development and Sustainment - Dr. Anne Hillegas, Applied Research Associates 
e Synthesis Panel**: Actions and Conference Takeaways 


Panelists: 


e Mr. Tom Wissink 
e Ms. Diana Echols, Director of Customer Service Technology, Bell Helicopter Textron 
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“Test & Evaluation in Support of Operational 
Suitability, Effectiveness and Sustainment of Deployed 
Systems” 
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Conference Ne 


Sponsored by the National 
Event # 7910 Defense Industrial Association Test 
March 12-15, 2007 & Evaluation Division In Cooperation 

with the NDIA Systems Engineering 


The Westin Resort and Logistics Divisions. The 


. Conference is Supported by the Office 
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Test & Evaluation (DOT&E) 


STRENGTH THROUGH INDUSTRY & TECHNOLOGY 


Conference Announcement 
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This National Test & Evaluation Conference is sponsored by NDIA’s Test & Evaluation Division in conjunction 
with NDIA Systems Engineering and Logistics Divisions. The critical issues being addressed and the potential 
solutions to them impact not only the T&E community but also cut across the disciplines involved with the 
confluence of Systems Engineering and Logistics. 
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of Iraq and Afghanistan. Some defense leaders contend that sustainment is only succeeding “on the backs of 
maintainers.” Ata previous NDIA event, a Service Secretary made clear that the nation’s fighting forces face 
severe challenges and deserve the best a nation can offer. He identified several “shared values” fundamental to 
making resource allocation decisions. The first two, 1) Operational Suitability and 2) Sustainment of deployed 
systems are the focus of this Conference. The solutions to these issues lie at the intersection of Test & 
Evaluation, Logistics and Systems Engineering. While Operational Suitability is one of the two most common 
metrics assessed in DoD Test & Evaluation, Sustainment of Combat Operations and Equipment gets hardly a 
mention. Yet, its cost exceeds that of Pentagon procurement, and hence must be considered within the Test 
& Evaluation context as well. 


Background 


The Department of Defense is undertaking an initiative to make sustainability a key performance parameter of 
new systems. In part, the initiative is in recognition of the fact that between 65-80 percent of the total life cycle 
cost occurs after procurement. Currently the inventory is maintained by about 678,000 DoD personnel and 
at a cost of about $72 billion per year. The Sustainability Key Performance Parameter will require mandatory 
reporting on Materiel Availability, Materiel Reliability, and Ownership Cost. System designs to optimize these 
will require new and focused emphasis on sustainability during system engineering, and test and evaluation 
with recognition of how the sustainment system will operate when the system is fielded and deployed ina 
combatant environment. 


Conference Objectives 


+ Discussion of T&E Metrics for Suitability and Sustainability (S&S) 
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- Test Planning to Assure Priority for Assessment of S&S 

- Planning and Implementing Sustainability as a KKP Effectively 

- Design Techniques Such as Condition-Based Maintenance and its T&E 
- Test Methodology 

- Testing for Realistic Estimates of Reliability 
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- Technologies to Reduce Life Cycle Cost 

- Field Test Data and Archiving 

- Feedback Sustainment Lessons to Improve Requirements, Programming T&E and Acquisition Process 


Following are members of the NDIA T&E Division Executive Committee 


- Mr. Joe Andrese, APG - Dr. Mark Kiemele, Air Academy Associates 
- Mr. Dennis Bely, ARL - Mr. Chuck Larson, SURVICE Engineering 

- Dr. Keith Bradley, LLNL - Mr. Jim O’Bryon, The O’Bryon Group, T&E 
- Mr. Britt Bray, DRC Corporation Division Chair 

- Dr. Paul Deitz, HRED - Dr. Ernest Seglie, DOT&E 

- Mr. Dick Dickson, Tybrin Corporation - Mr. Jack Sheehan, FCSCTO 

- Mr. Russ Hauck, Simulation Info Systems - Dr. Lowell Tonnessen, IDA 

- Dr. Anne Hillegas, ARA Corporation - Dr. Juan Vitali, ATEC 

- Mr. John Illgen, Northrop Grumman - Mr. Tom Wissink, Lockheed Martin 

- RADM Bert Johnston, USN (Ret), Wyle Labs - Mr. Bill Yeakel, ORSA Corporation 
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NDIA Test & Evaluation Division 


The Test & Evaluation Division was initially established under the auspices of the American Defense 
Preparedness Association (ADPA). With the merger of the National Security Industrial Association 
and the American Defense Preparedness Association on March 1, 1997, the T&E Division and its 
responsibilities were transferred to the National Defense Industrial Association. It is one of the 32 
divisions of the Association. 


NDIA TEST & EVALUATION DIVISION OBJECTIVES 

NDIA’s principal missions are to improve weapons technology, improve defense management, and 
maintain a strong science-industry-defense team continually responsive to all needs of the research, 
development, test & evaluation, production, logistics and management phase of national preparedness. 
It provides a forum for the exchange of ideas and information between its members and government 
agencies through a network of divisions, chapters, national and local meetings, and conferences and 
visits to the Department of Defense installations. Consistent with, and in furtherance of, the By-Laws 
and policies of the NDIA, the objectives of the T&E Division with respect to those government elements 
appropriate to its field of interest are: 


Ensure the continuation, responsiveness, and where necessary, the revitalization of the industrial base 
to support our national security by: 


Maintaining effective liaison with the Department of Defense, Executive and Legislative Branches 
of the Government, and other governmental departments and agencies to apprise appropriate 
representatives of these organizations of T&E Division activities and keep abreast of current and 
future developments. 


Reviewing, evaluating, and providing recommendations with regard to government policies, 
practices, directives, and specifications including pending legislation concerning life cycle logistic 
support, supply chain integration, strategic and tactical mobility and business processes. 


- Serving as a catalyst for community dialogue among T&E communities within government, industry 
and academia as well as provide an overarching means for dynamic interchanges of information with 
other functional areas. 


Sponsoring government/industry meetings, seminars and symposia to provide effective 
communications between government and industry relative to management activities for which the 
T&E Division is responsible under this matter. 


Maintaining viable liaison with other divisions of NDIA, other associations, and national-level 
industry coalitions for the purpose of exchanging information of mutual interest and coordinating 
and cooperating on activities requiring joint participation. 


- Developing strategic directions effecting T&E partnerships between national security industries and 

government and to formulate proposals for national policies, addressing their impact on the national 

security industrial base. 

- Establishing and maintaining a productive proactive dialogue with the national news media. 
Fostering learning of the basics and advanced concepts and techniques of T&E within the 

defense industry including the cross-pollination of commercial approaches to T&E for the defense 


community. 


Commemorating and recognizing services rendered by individuals, companies, and government 
agencies in meritorious acts related to activities enumerated above. 


Register Now! www.ndia.org/meetings/7910 =3°= 


Monday, March 12, 2007 


10:00 AM - Conference Registration 
5:00 PM 


1:30 - 3:00 PM CONFERENCE TUTORIALS 
Tutorial Sessions are provided free of charge to all those registering for the 
Conference. Your attendance at these valuable tutorials is encouraged. 


TUTORIALS: Sessions T1 - T4 


Tutorial Tl. Innovative Acquisition Methodologies in Support of Test & Evaluation 
Objectives - Mr. Jeffrey Manthos, Program Analyst, GSA 


Tutorial T2. The Critical Link Between Test & Evaluation and Modeling & Simulation 
- Dr. Mark Kiemele, President and Co-Founder, Air Academy Associates 


Tutorial T3. Reliability Growth Applied to Defense Issues- Dr. Larry Crow, President, 
Crow Reliability Resources, Inc. 


Tutorial T4. DoD Acquisition Policy on Suitability KPP and Related Issues - Dr. Paul 
Alfieri, Director of Research, DAU 


3:00 - 3:15 PM _ Tutorial Afternoon Break 


3:15 - 5:00 PM TUTORIALS: Sessions T5 - T8 (Continued Sessions) 


Tutorial T5. Innovative Acquisition Methodologies in Support of Test & Evaluation 
Objectives - Mr. Jeffrey Manthos, Program Analyst, GSA 


Tutorial T6. The Critical Link Between Test & Evaluation and Modeling & Simulation 
- Dr. Mark Kiemele, President and Co-Founder, Air Academy Associates 


Tutorial T7. Reliability Growth Applied to Defense Issues- Dr. Larry Crow, President, 
Crow Reliability Resources, Inc. 


Tutorial T8. DoD Acquisition Policy on Suitability KPP and Related Issues - Dr. Paul 
Alfieri, Director of Research, DAU 


5:00 - 6:30 PM ___ Evening Reception/Displays Open 


6:30 PM Display Area Closed for the Day 


Your attention please: Survey sheets will be given to every conference registrant upon arrival. The survey 
sheets are intended to gather your thoughts and ideas regarding this NDIA T&E Conference. Registrants 
are requested to thoughtfully complete these surveys and return them to the NDIA desk prior to their 
departure. The results of these surveys will serve as a basis in planning for future conferences. 


——_aed = 


The Westin Resort, Hilton Head Island, SC 
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Tuesday, March 13, 2007 


7:00 AM - 8:00 AM__ Continental Breakfast in the Display Area 


7:00 AM- 5:00 PM_ Conference Registration 


_ — Session A - 
Session Chair: Mr. James O’Bryon 
8:00 AM Call to Order and Remarks - Mr. Samuel Campagna, NDIA 
8:05 AM Tribute to Our Nation, National Anthem 
8:10 AM Welcome and Conference Introductory Remarks- Mr. James O’Bryon, Chairman, NDIA Test & 


Evaluation Division 


8:20 AM 
Conference Keynote Address: Assuring Quality and Sustainability in a Competitive 
Market - Mr. Keith E. Williams, President & CEO, Underwriters Laboratories 
8:55 AM 
Government Keynote Address: Getting Our Arms Around Motivating and Testing 
for Sustainability - Honorable Charles E. McQueary, Director, Operational 
Test & Evaluation, OSD 
9:30 AM 
Military Perspectives Keynote Address: Gen Larry D. Welch, USAF (Ret), 
President, IDA, Former Chief of Staff, USAF 
10:05 AM Morning Break & Networking in the Display Area 
— Session B - 
Session Chair: Mr. James O’Bryon 
10:35 AM OTA’s Roundtable: Military Perspectives on Testing for Sustainability and Life Cycle Costing 
Chair: MG James Myles, USA, Commanding General ATEC 
Panelist: RDML William McCarthy, USN, Commander OPTEVFOR 


Maj Gen Robin Scott, USAF, Commander AFOTEC 
COL Michael Bohn, USMC, Commander MCOTEA 


12:15 PM Luncheon 


— Session C - 
Session Chair: Dr. Mark Kiemele 
1:30 PM Statistically-Based Test Optimization Strategies - Dr. Neal Mackertich, Raytheon Integrated 
Defense Systems 
2:10 PM Report from the NDIA Industrial Committee on Operational Test & Evaluation (ICOTE) - Mr. 


Larry Graviss, Vice President, Jacobs Technology, Inc. 


2:20 PM Afternoon Break & Networking in the Displays Area 


Register Now! www.ndia.org/meetings/7910 SS 


Tuesday, March 13, 2007 - Concurrent Focus Sessions 
— Session D - 
2:50 - 5:30 PM Planning & Implementing Systems Engineering/T&E to Address 
Sustainability 


FS D1.1 T&E for Verifying Technology Development and Maturation - Mr. Chris 
DiPetto, Deputy Director for DT&E, OUSD (AT&L)/SSE 


FS D1.2 Early Application of Computer Program Systems Integration, Test and 
Performance Measurement - Mr. Thomas Sobieralski, CSC 


FS D1.3. Using Technology Readiness Assessments (TRA’s) to Assess the Maturity of 
Life Cycle Related Technologies - Dr. Jay Mandelbaum, IDA 


FS D1.4 Results of FY 2006 Test Resources Management Center Range Encroachment 
Survey Provided to the August 2006 TRMC Infrastructure Review - Mr. William Egan, 
IDA 
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Environment Test Capability JMETC) - Mr. Gene Hudgins, BAE Systems 


FS D1.6 TRACER: A Tradespace Analysis Framework - Mr. Jerry Belcher, ASARDA 


2:50 - 5:30 PM Reducing Total Cost of Ownership & Realistic Estimates of Reliability 
FS D2.1 Reduction of Total Ownership Costs (R-TOC) and Value Engineering - Mr. 
David Erickson, USD (AT&L) 


FS D2.2 Testing of the GPS SAASM End-to-End Functionality on Operational Weapon 
Systems Without the Availability of Signal in Space - Mr. Jim Killian, 46 Test Squadron, 
Holloman AFB 


FS D2.3 The Costs of Unsuitability and Benefits of Building in Reliability, Availability 
and Maintainability- Dr. Ernest Seglie, DOT&E 


FS D2.4 A Successfully Implemented Coordinated Subsystem Reliability Growth 
Planning Approach- Mr. Louis Chenard, Bombardier 
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James A. Forbes, LMI 


FS D2.6 Using Mission Essential Task Lists as a Basis for Mission-Based Operational 
Test Planning - Mr. James Carpenter, AVW Technologies 


2:50 - 5:30 PM Test Technology 


FS D3.1 Hazard Assessment of the SM-3 Block IA Missile — Mr. William Houchins, 
NSWC Dahlgren 


FS D3.2 The Four Element Framework: An Integrated Test & Evaluation Strategy - Mr. 
Christopher Wilcox, ATEC, APG 


FS D3.3 Operability Testing for High Availability Systems- Mr. Edward Beck, CSC 
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Tests: M&S vs. Raw Data, What is Needed? - Mr. Jerome Perkins, SAIC 


FS D3.5 746 Test Squadron Testing of the Latest Embedded GPS/INS (EGI) Hybrid 
Navigation System for the F-16 Fighting Falcon - Mr. Jim Killian, 46 Test Squadron, 
Holloman AFB 


FS D3.6 Qualification Testing of the EFSS 120mm Mortar Ammunition - Mr. Patrick 
Freemyers, NSWC Dahlgren 
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Enhance System Sustainability - Dr. Don Gaver, NPS 


5:30 PM Afternoon Sessions Closes/Display Area Closes for the Day 
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Tuesday, March 13, 2007 - Evening Activities 


6:30 PM NDIA Honors Banquet 


Presentation of the Walter W. Hollis Award for Lifetime Achievement in Test & Evaluation 
This event is one of the highlights of our NDIA T&E Annual Conference. This year, we again take time 
to honor an individual who has demonstrated a lifetime commitment and contribution to the honest and 
realistic conduct of Test & Evaluation in support of the defense of the United States. This year, we are proud 
to honor such a person: Mr. James F. O’Bryon. 


Guest of Honor - Mr. James F. O’Bryon 


Mr. James O’Bryon’s career began at IBM’s R&D Center in NY while still in 
undergraduate school. Subsequent to this, he took a position in the Actuarial 
Department of New York Life. The Army drafted him in December 1964 and this 
moved him into the T&E world. He held RDT&E positions at the Aberdeen Proving 
Ground from 1965-1986. In 1986, he accepted the position as the first Director, 
Live Fire Testing in the Pentagon and retired from the Pentagon, November 2001 as 
Dep. Dir. OT&E/LFT&E, a Deputy Assistant Secretary of Defense position. Jim has 
an undergraduate math degree and graduate degrees from GWU in Ops Research 
and from MIT in Electrical Engineering. 


Jim has chaired the NDIA T&E Division for 14 years and chaired dozens of national 
and international T&E Conferences. He has taught T&E at NDU, ICAF, NPS, University 
of Texas and other places for nearly 20 years, authored over 90 technical reports, 
and 80 open literature articles on T&E and related issues. He has just completed 
an 800 page book on Lessons Learned from Live Fire Testing and has testified 
before Congress on several occasions on T&E and Aviation Security. He has received the ADPA Gold Medal, 
the NDIA Gold Medal, the Arthur Stein Award for Excellence in Live Fire Testing and numerous DoD Awards 
including two personally presented by two different Secretaries of Defense. He now chairs a Committee of 
the National Academy of Sciences as well as serving as a consultant to several commercial and not-for-profit 
institutions. 


Previous Recipients 

2006: RADM Bert Johnston, USN (Ret), Wyle Laboratories 2002: Mr. G. Thomas Castino, Underwriters Laboratories, Inc. 
2005: Hon Thomas Christie, DOT&E, OSD 2001: Hon Philip E. Coyle, Ill, DOT&E, OSD 

2004: Dr. Marion L. Williams, Headquarters AFOTEC 2000: Mr. Walter W. Hollis, Department of the Army 

2003: Mr. James W. Fasig, Aberdeen Test Center 


Guest Speaker - Mr. Raymond Daddazio 


It is also our special privilege to welcome as our guest 
speaker, Mr. Raymond Daddazio, President, Weidlinger 
Associates, New York, NY. September 11, 2001 is a day 
we will always remember and the scenes of the World 
Trade Center collapsing is forever etched in our minds. 


Many were shocked not only by the aircraft attack against 
the buildings, but also shocked to see that these buildings 
collapsed from the attack, resulting in a huge loss of life 
and property. Weidlinger Associates was called upon to 
lead the efforts to assess what and how the WTC collapsed 
and also how to mitigate a repeat of such a failure if future 
attacks occur on other major structures. Mr. Daddazio 
will share with us his insights into their analysis and 
recommendations. 


Note: Guests wishing to attend with registrants are welcome to attend this banquet. However, additional tickets must be purchased 
at the NDIA desk prior to the banquet. 


Register Now! www.ndia.org/meetings/7910 == 


Wednesday, March 14, 2007 


7:00 Ava Continental Breakfast in the Display Area 
8:00A 

7:00 an = Conference Registration 

5:00 P 


- Session 
Session Chair: Mr. William Yeakel 


8:00 AM Call to Order and Remarks - Mr. Samuel Campagna, NDIA 
8:05 AM T&E Study Results—- GEN David Maddox, USA (Ret), Former Commander-in-Chief, US 
Forces Europe 
8:35 AM Warfighting in a Climate Warming World: The Expectations, the Options, and the 
Solutions - Hon Philip Coyle, III, Former Director, OT&E, OSD 
9:05 AM Reliability Growth as Applied to Military Systems - Dr. Larry Crow, President, 
Crow Reliability Resources, Inc. 
9:35 AM Overcoming the Lowest Bidder with Quality Products in Government Purchases - Mr. 
William Britz, President, Federal Program Management 
10:05 AM Morning Break & Networking in the Display Area 
— Session F - 
Session Chair: Mr. Dick Dickson 
10:35 AM Development, Test & Evaluation of Unmanned Ground Vehicles/Robots, VADM 
Joseph Dyer, USN (Ret), President and General Manager, iRobot Corporation 
11:05 AM NDIA Update Snapshot - MG Barry Bates, USA (Ret), Vice President, Operations, NDIA 
11:15 AM Sustainability KPP: Industry Weighs in on Bidding for It and Building for It - Mr. John 


Stoddart, President, Defense Business, Oshkosh Truck 
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Wednesday, March 14, 2007 


1:15 PM 


1:45 PM 


- Session G 
Session Chair: Mr. John Illgen 


Coping with Reliability, Sustainability, and Life Cycle Costs at DHS - Dr. George 
Zarur, Science Advisor, Science and Technology, DHS 


DT&E Panel: Systems Engineering and DT&E for System Sustainability 


Chair: Col Richard Stuckey, USAF, Principal Assistant Deputy Director for 
DT&E, OUSD (AT&L)/SSE 
Panelists: Dr. David Jerome, Deputy Director of Air, Space and Information 


Operations, AF Materiel Command 

Mr. Brian Simmons, Deputy to the Commander/Technical Director, 
Army DTC 

Mr. Richard Shubert, VP and Chief Engineering, Lockheed Martin 
Integrated Systems and Solutions 

Mr. Ray Lytle, Director, Life Cycle Engineering, Raytheon Missile 
Systems 


Wednesday, March 14, 2007 - Concurrent Focus Sessions 


3215-= 5:15 PM 


Focus 
Session H4 


Session Chair: 
Dr. Lowell 
Tonnessen 


3:15 - 5:15 PM 


Focus 
Session H5 


Session Chair: 
Mr. Dick 
Dickson 


3215.= 5215 PM 


Focus 
Session H6 


Session Chair: 
Dr. Anne 
Hillegas 


5:15 PM 


— Session H - 
Test Methodology 


FS H4.1 Revisiting Quantitative Methods for Evaluating Training Programs for Systems 
Undergoing OT - Dr. Christopher Hekimian, SAIC 


FS H4.2 Producing Anywhere, Anytime Test, Evaluation and Diagnostics Capable 
Products to Eliminate the T&E Logistic Burden - Mr. Duane Lowenstein, Agilent 
Technologies 


FS H4.3 Ship Suitability Testing - Preparing for the Future - Mr. Ronald Harney, 
NAVAIR, Patuxent River 


FS H4.4 Innovative Techniques to Solve Measurement and Detection Problems - Dr. 
Michael Slocum, Air Academy Associates 


Test Planning 
FS H5.1 A Continuum of Testing for Military Systems - Dr. Patricia Jacobs, NPS 


FS H5.2 CTEC (Collaborative Test & Evaluation Capability) and the Event Streaming 
Media System - Mr. Reid Johnson, NUWC Keyport 


FS H5.3 Integrated Weapons System Testing - Ms. Laura DeSimone, NSWC Dahlgren 


FS H5.4 Estimating Durations and Trials to Success in Test Trial Programs - Dr. Danny 
Hughes, LMI Government Consulting 


FS H5.5 Operational Testing of Procedures is Critical to Effective Use/Integration of 
Emerging Capabilities:Report on Joint Space Control Operations - Mr. John Marrs, JT&E 


Technology to Reduce Life Cycle Cost 
FS H6.1 T&E of Electromagnetic Railguns - Mr. Thomas Boucher, NSWC Dahlgren 


FS H6.2 Cost Efficient Risk Management Through Integrated T&E Throughout the 
Systems Engineering Life-Cycle - Mr. Joseph Tribble, AVW Technologies 


FS H6.3 Sustainment of the Deployed Navy Munitions Inventory Through Continuous 
Quality Evaluation Test & Evaluation - Mr. Jeffrey Johnson, Navy Indianhead 


Display Area Closes for the Day 


Register Now! www.ndia.org/meetings/7910 = 95 


Thursday, March 15, 2007 


Continental Breakfast in the Display Area 


Conference Registration 


— Session | - 
Session Chair: Mr. John Illgen 


Call to Order and Remarks - Mr. Samuel Campagna, NDIA 
Suitability - At What Cost? - Dr. Paul Alfieri, DAU 


Service T&E Executives Speak 
Chair: Mr. Walt Hollis, Former DUSA/OR 
Panelists: Mr. Dave Duma, Principal Deputy, DOT&E 
Dr. Hank Dubin, Army T&E 
Mr. Jack Manclark, AF/TE 
Mr. Steven Whitehead, OPTEVFOR 


9:05 AM The Costs of Unsuitability and Benefits of Building in Reliability, Availability and 
Maintainability - Dr. Ernest Seglie, DOT&E 


9:30 AM Army T&E - Providing Essential Information to an Army at War - Mr. Brian Simmons, 
Army DTC 


10:00 AM Morning Break & Networking in the Display Area (Last Opportunity to Visit Displays) 


— Session J - 
Session Chair: Mr. William Yeakel 
10:20 AM Reliability-Based Design, Development and Sustainment - Dr. Anne Hillegas, Applied 
Research Associates 


11:15 AM Synthesis Panel”: Actions and Conference Takeaways 
Chair: Dr. Ernest Seglie, Science Advisor, DOT&E 
Panelists: Ms. Diana Echols, Director of Customer Service Technology, 
Bell Helicopter Textron 
Mr. Steven Whitehead, Technical Director, OPTEVFOR 
Mr. Stephen Daly, DDOT&E, Conventional Systems 
Mr. Chris DiPetto, Deputy Director for DT&E, OUSD (AT&L)/SSE 


“The discussion of this panel on the overall thrust of the Conference will 
serve as the basis of a White Paper and an overview article to be published 
in NDIA’s National Defense Magazine. 


11:55 AM Conference Closing Remarks - Mr. James O’Bryon, Chairman, NDIA Test & Evaluation 
Division 


Note: please be sure to return your completed survey forms to the NDIA 
registration desk 


“The Department of Defense finds this event meets the minimum regulatory standards for attendance by DoD employees. This finding does 
not constitute a blanket approval or endorsement for attendance. Individual DoD component commands or organizations are responsible for 
approving attendance of its DoD employees based on mission requirements and DoD regulations.“ 
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The Westin Resort - Hilton Head Island 
Two Grasslawn Avenue 
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Phone : (843) 681-4000 
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Please call the hotel directly to make your reservation. 

In order to ensure the discounted NDIA rate, make your 
reservations early and identify yourself as an attendee of 
the NDIA Test & Evaluation Conference. Rooms will not 
be held after February 19, 2007 and may sell out before 
then. Rates are also subject to increase after this date. 


Government Rate*: $120.00 
Industry Rate: $159.00 


*or the prevailing government per diem rate at the time of the Conference. The government per diem rate is available only 
to active duty military and civilian government employees. ID will be required upon check-in. Retired military or government 


Acie i kiee-cerelamiaicelaaarclarele 


Early Regular Onsite 
Before 1/29/07 1/29/07- 2/26/07 _—_ after 2/26/07 


Government/ Allied / 
Academia $590 $650 $715 


Industry NDIA 
Member $690 $760 $835 
Industry Non-NDIA 
Member $730 $810 $890 


To register online for this Conference, please visit the following link: 

http: //www.ndia.org/meetings/7910. Online registration will close on February 26, 2007. You must 
register onsite after this date. You can also download the registration form from the NDIA website or 
complete the form contained in this brochure. Fax the completed form to (703) 522-1885 or mail to: 
Event #7910, National Defense Industrial Association, 2111 Wilson Boulevard, Suite 400, Arlington, 
VA 22201-3061. Please do not fax or mail registration forms after February 26 , 2007. You will need 
to register onsite after this date. Payment must be made at the time of registration. Registrations will 
not be taken over the phone. 


CANCELLATIONS REMINDER: Cancellations received before January 29, 2007 will receive a full refund. 
Cancellations received from January 29, 2007 until February 26, 2007 will receive a refund minus a 
cancellation fee of $75.00. Refunds will NOT be given for cancellations received after February 26, 
2007. Substitutions are welcome in lieu of cancellations. 


Register Now! www.ndida.org/Mmeetings/79iG = jib} 
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NDIA Tester of the Year Awards 


The Annual NDIA Tester of the Year Awards will be presented during the Conference honoring those 
representatives from industry, government civilians and military personnel from the Army, Navy, Air Force, 
Marine Corps and OSD for outstanding achievement. 


Displays 


NDIA TanVic-om ole meen ocelanlanlolal(ertu-m\olo) melael-lalp4-idce)armer-tey-le)ilid(-t-w-l manly aol gem ValalerclMnt-ldlolar-lma-imcm a- OT lelole 
Forum. An area will be available for the set-up of displays to demonstrate your company/organization’s 
unique products and related capabilities. Displays will be tabletop or “pop-up” style. Allocated display 
space will be 10’ wide by 6’ deep. Pipe and drape will not be provided. The display registration form is 
available on the Conference website: www.ndia.org/meetings/7910. For additional information on displays, 
please contact Carissa Mirasol, Meeting Planner at cmirasol@ndia.org or (703) 247-2588. Cost to display: 
$1000.00. This fee includes one complimentary registration. Please refer to the Displayer Registration Form 
on page 15 of this brochure. 


magelaatolare)ar-lim cl aeal-1ecvalleme@)e)elelauelaliarsts 


Take g-t-WX-MZel¥) eo) eal ey-lan mel me) ger-lalp4-hale)aiecn->.4olek10 1 a-e-l mda im 0) a-leal(-)m-\A-1 ale oa ol-xe) anil alem- 6 Gol ali-la-1a(e-8 aaelearovaleyar-l 
Partner. A Promotional Partnership ($5,000) will add your company name/logo to the back cover of the onsite 
brochure as well as main platform recognition throughout the Conference, signage at all events including the 
rol ol-1ali ale mm a=1e-) o)d ce) ae- late Me- Wms Hol Orn) 0) a0 Mo) cel-Valp2- luce) ame (-\Yog) o)a(o) aml aim dal-in Go) aii-la-la(e-mr-\el-1a(er- Mumm ©) dalam age) aatelece)ar-l| 
Partnership levels are available. For more information, please contact Sam Campagna at (703) 247-2544 or 
iFM) aati mre lamcrer-laaley-Celar-\Cualelt-Melaer 


Kel-Tandiaxerctdrelamsy-(e (ert 


During Conference Registration and check-in, each participant will be issued an identification badge. Please 
be prepared to present a picture ID. Badges must be worn at all Conference functions. 


(@olaicsalaleew- Nadia 


Appropriate dress for this Conference is coat and tie for civilians and Class A uniform for military 
personnel. 


Attendee Roster 


An attendee roster will be distributed at the Conference. In order for your name to appear in the Conference 
attendee roster, you MUST register by February 26, 2007. There will be NO additional or updated versions. 


Conference Proceedings 


Proceedings will be available on the web through the Defense Technical Information Center (DTIC), and will 
be available two to three weeks after the Conference. You will receive notification via email that proceedings 
are posted and available on the web. 
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National Defense Magazine 


Advertise in National Defense and increase your company’s exposure at this Conference. National Defense 
will be distributed to attendees of this Conference as well as other NDIA events. For more information contact 
ID) fate ake] ake) adie-lul @A0 ks) WV2- Weer aoL amo) anVAT-M-e0r-lim-hemele)Kelaveladi@ualelr-Melnem 


Special Needs 
NDIA supports the Americans with Disabilities Act of 1990. Attendees with special needs should call (703) 


522-1820 and refer to the 23rd Annual National Test & Evaluation Conference, Event #7910, prior to February 
26, 2007. 


Inquiries? 


For more information on the technical program, please contact the Conference Chairman, Mr. James O’Bryon 
at (443) 528-2711 or via email at jamesobryon@obryongroup.com. 


For questions on the Conference, please visit the Conference website at www.ndia.org/meetings/7910 or 
contact Carissa Mirasol, Meeting Planner at (703) 247-2588 or via email at cmirasol@ndia.org. 


Don’t miss out on the upcoming 
Live Fire Test & 
Evaluation Conference 
June 25-28, 2007 


For more information, please visit: 
www.ndia.org/meetings/7390 


For inquiries, please contact Heather Horan, 
WW/KeXcha pale pm at-Valatclam-tem alavele-talCualelt-Melaeme)ar-lt 
(703) 247-2570 


Register Now! www.ndia.org/meetings/7910 == 


23rd Annual National Test & Evaluation 
Conference 
Westin Resort ¢ Hilton Head Island, SC 
March 12-15, 2007 ° Event #7910 


National Defense Industrial Association 
2111 Wilson Boulevard, Suite 400 
Arlington, VA 22201-3061 

(703) 522-1820 « (703) 522-1885 fax 
www.ndia.org 


NATIONAL DEFENSE INDUSTRIAL ASSOCIATION 


Lh TT ai hy 


STRENGTH THROUGH INDUSTRY & TECHNOLOGY 


3 Ways to sign up: 1. Online with a credit card at www.ndia.org 
2. By fax with a credit card — Fax: 703-522-1885 


3. By mail with a check or credit card 


NDIA Master ID/Membership # 


(if known—hint: on mailing label above your name) 


Social Security # 
(last 4 digits — optional) 


By completing the following, you help 
us understand who is attending our 
meetings. 


QO Address change needed 


Primary Occupational 
Classification. Check ONE. 


; QA. Defense Business/Industry 
Prefix OB. R&D/Laboratories 
(e.g. RADM, COL, Mr., Ms., Dr., etc.) Oc. Army 
Name First MI Last OD. Navy 
QE. Air Force 
Military Affiliation Nickname QF. Marine Corps 
(e.g. USMC, USA (Ret.) etc.) (for Meeting Badges) OG. Coast Guard 
; Q H. DOD/MOD Civilian 
Title Ql. Gov't Civilian (Non-DOD/ 
O izati MOD) 
igenizeuon QJ. Trade/Professional Assn. 
Street Address Q K. Educator/Academia 
QOL. Professional Services 
Address (Suite, PO Box, Mail Stop, Building, etc.) OM. Non-Defense Business 
ON. Other 
City State Zip Country 
Current Job/Title/Position. 
Phone ext. Fax Check ONE. 
E-Mail DA. Senior Executive 
OB. Executive 
Signature* Date QC. Manager 
QD. Engineer/Scientist 
Preferred way to receive information QE. Professor/Instructor/Librarian 
Conference information O address above Q Alternate (print address below) Q E-mail QF. Ambassador/Attaché 
Q G. Legislator/Legislative Aide 
Subscriptions Q address above QO Alternate (print address below) OH. General/Admiral 
QI. Colonel/Navy Captain 
Alternate Street Address OJ. Lieutenant Colonel/ 
Alternate Address (Suite, PO Box, Mail Stop, Building, etc.) Commander/Major/ 
Lieutenant Commander 
City State _ Zip Country QO K. Captain/Lieutenant/Ensign 
QL. Enlisted Military 
* By your signature above you consent to receive communications sent by or on behalf of NDIA, its Chapters, Benet 
Divisions and affiliates (NTSA, AFEI, PSA, NCWG, WID) via regular mail, e-mail, telephone, or fax. NDIA, its Year of birth 
Chapters, Divisions and affiliates do not sell data to vendors or other companies. (Optional) 
Registration Fees Payment Options 
Early Regular Late a Check (payable to NDIA) 
(Before 1/29/07) (From 1/29/07 - 2/26/07) __ (After 2/26/07) 4 Cash - 
Gaveament a Government PO/Training Form # 
Academia/Allied: a $590 a $650 a $715 a VISA 
a MasterCard 
Industry NDIA 7 tinge tis (oes 
Member: a $690 a $760 oO $835 . : 
$ $ $ If paying by credit card, you may return by fax to (703) 522-1885. 
Credit Card Number 
Industry Non-NDIA 
Member, $730 a $810 aso | LIU 


Cancellations received before January 29, 2007 will receive a full refund. 
Cancellations received from January 29, 2007 to February 26, 2007 will 
receive a refund minus a cancellation fee of $75. No refunds for cancellations 
received after February 26, 2007. Substitutions are welcome in lieu of 
cancellations! 


Questions? Contact Meeting Planner, Carissa Mirasol 
(703) 247-2588 email: cmirasol@ndia.org 


Mail to: NDIA, Event #7910 
2111 Wilson Boulevard, Suite 400 
Arlington, VA 22201 

Fax to: (703) 522-1885 


00,00 


Exp. date 


Signature Date 


23rd Annual National Test & Evaluation Forum 
March 12-15, 2007 
Westin Resort —- Hilton Head Island - Hilton Head Island, SC 
Registration for Displays - Event #7910 


Name 

Title 

Company Name 

Division/Dept. 

Address 

City/State/Zip 
Phone éFax 
E-mail 


Display Requirements: 

All displays must be of the simple table-top/pop-up style standards. Space per display shall not exceed 10 ft. wide 
by 6 ft. deep. Minimal hardware to be utilized (computer systems for demonstrations are acceptable). No formal 
decorating company is involved. Companies must bring their own displays and plan to do their own set-up. Standard 
2.5 x 6 ft. draped folding tables and chair will be provided for each display space. No other props or setups (pipe & 
drape, plants, etc.) will be utilized. 


Display Hours: 
Displays are to be set up by 4:00 PM March 12, 2007. Displays must be removed by 12:00 noon March 15, 2007. 


Cost: Displays (includes complimentary registration for one displayer and electrical hook-up): $1000 


Display Rules & Regulations: 

1. If NDIA should be prevented from holding the conference for any reason beyond NDIA’s control (such as, but not 
limited to, damage to the building, riots, strikes, acts of government, or acts of God) or if a displayer cannot occupy 
the assigned display space due to reasons beyond NDIA’s control, then NDIA has the right to cancel the conference or 
any part thereof, with no further liability to the displayer other than a refund of display space fee, less a proportionate 
share of the conference cost incurred. 

2. Neither the management of the host facility nor NDIA shall be liable for the damages, loss or destruction to the displays 
by reason of fire, theft, accident or other destructive causes. Displayer shall lease space at his sole risk. Neither the 
management of the host facility, NDIA nor any of their agents, servants or employees will be accountable or liable for 
accidents to displayers, their agents or employees. 

3. The displayer shall be liable to the host facility and/or NDIA for any damage to the building and/or the furniture and 
fixtures contained therein which shall occur through acts or omissions of the displayer. 

4. Displayer assumes the entire responsibility and hereby agrees to protect, indemnify, defend and hold harmless 
NDIA, the host facility, their officers, employees, and agents against all claims, losses and damages to persons and 
property, governmental charges or fines, and attorney’s fees arising out of or caused by displayer’s installation, removal, 
maintenance, occupancy or use of the display premises or any part thereof, including any outside display areas. 

5. Displayer acknowledges that NDIA does not maintain and is not responsible for obtaining insurance covering 
displayer’s property. Displayers are advised to obtain business interruption and property damage and loss insurance 
to cover such occurrences. 


Send this form with payment for display to: 
Carissa Mirasol, National Defense Industrial Association, 2111 Wilson Boulevard, Suite 400, Arlington, VA 22201 
Phone: (703) 247-2588, Fax: (703) 522-1885, E-mail: cmirasol@ndia.org 


Deadline for sign-up is February 26, 2007. Make checks payable to NDIA - Event # 7910. 


OCheck (payable to NDIA - Event #7910) OVisa ODiner’s Club OiMastercard OAmex 


Credit Card # 


NATIONAL DEFENSE INDUSTRIAL ASSOCIATION 
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STRENGTH THROUGH INDUSTRY & TECHNOLOGY 


2111 Wilson Blvd., Suite 400 
Arlington, VA 22201 

p (703) 522-1820 

f (703) 522-1885 


Thank you to our 
Conference 
Promotional Partner: 


AIR 
ACADEMY 
ASSOCIATES 


If you are interested in becoming a Promotional 
Partner for this Conference, please visit: 
www.ndia.org/meetings/7910. 
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Test & Evaluation Conference 


“Test & Evaluation in Support of Operational 
Suitability, Effectiveness and Sustainment of Deployed 
Systems” 
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Conference Ne 


Sponsored by the National 
Event # 7910 Defense Industrial Association Test 
March 12-15, 2007 & Evaluation Division In Cooperation 

with the NDIA Systems Engineering 


The Westin Resort and Logistics Divisions. The 


. Conference is Supported by the Office 
Hilton Head Island, SC of the Under Secretary of Defense 
(NUK Valo id al-in DJ n-Veiko) am ©) el-1e-luro)arel 


Test & Evaluation (DOT&E) 


STRENGTH THROUGH INDUSTRY & TECHNOLOGY 


Conference Announcement 


-Nalateletavestaakevaie 


This National Test & Evaluation Conference is sponsored by NDIA’s Test & Evaluation Division in conjunction 
with NDIA Systems Engineering and Logistics Divisions. The critical issues being addressed and the potential 
solutions to them impact not only the T&E community but also cut across the disciplines involved with the 
confluence of Systems Engineering and Logistics. 


BW al-e-1)]lia’Ae) MUN Kee) aa] oy-ten Ke) ceKmcoMEK I e-llamaal-limy Veale) alaialemer-Vey-leyilinvmiwex-dialemnckae-lemol-li Na lamaal-m@o)ealey-Vmdal-t-Nd-1as) 
of Iraq and Afghanistan. Some defense leaders contend that sustainment is only succeeding “on the backs of 
maintainers.” Ata previous NDIA event, a Service Secretary made clear that the nation’s fighting forces face 
severe challenges and deserve the best a nation can offer. He identified several “shared values” fundamental to 
making resource allocation decisions. The first two, 1) Operational Suitability and 2) Sustainment of deployed 
systems are the focus of this Conference. The solutions to these issues lie at the intersection of Test & 
Evaluation, Logistics and Systems Engineering. While Operational Suitability is one of the two most common 
metrics assessed in DoD Test & Evaluation, Sustainment of Combat Operations and Equipment gets hardly a 
mention. Yet, its cost exceeds that of Pentagon procurement, and hence must be considered within the Test 
& Evaluation context as well. 


Background 


The Department of Defense is undertaking an initiative to make sustainability a key performance parameter of 
new systems. In part, the initiative is in recognition of the fact that between 65-80 percent of the total life cycle 
cost occurs after procurement. Currently the inventory is maintained by about 678,000 DoD personnel and 
at a cost of about $72 billion per year. The Sustainability Key Performance Parameter will require mandatory 
reporting on Materiel Availability, Materiel Reliability, and Ownership Cost. System designs to optimize these 
will require new and focused emphasis on sustainability during system engineering, and test and evaluation 
with recognition of how the sustainment system will operate when the system is fielded and deployed ina 
combatant environment. 


Conference Objectives 


+ Discussion of T&E Metrics for Suitability and Sustainability (S&S) 

Pa sXe [bel fae pm Moye=V im ©) /Za1-1 a" al] om @Xoy-} mor-Vale Xe) (Wo) aml Kc'd sens) Si- Vale I ole] aT ot 

- Test Planning to Assure Priority for Assessment of S&S 

- Planning and Implementing Sustainability as a KKP Effectively 

- Design Techniques Such as Condition-Based Maintenance and its T&E 
- Test Methodology 

- Testing for Realistic Estimates of Reliability 

Pa X-Yo) Kel] ale pm Kol-| im @cos-] aro) mm O)Zal-1a1 0119) 

- Technologies to Reduce Life Cycle Cost 

- Field Test Data and Archiving 

- Feedback Sustainment Lessons to Improve Requirements, Programming T&E and Acquisition Process 


Following are members of the NDIA T&E Division Executive Committee 


- Mr. Joe Andrese, APG - Dr. Mark Kiemele, Air Academy Associates 
- Mr. Dennis Bely, ARL - Mr. Chuck Larson, SURVICE Engineering 

- Dr. Keith Bradley, LLNL - Mr. Jim O’Bryon, The O’Bryon Group, T&E 
- Mr. Britt Bray, DRC Corporation Division Chair 

- Dr. Paul Deitz, HRED - Dr. Ernest Seglie, DOT&E 

- Mr. Dick Dickson, Tybrin Corporation - Mr. Jack Sheehan, FCSCTO 

- Mr. Russ Hauck, Simulation Info Systems - Dr. Lowell Tonnessen, IDA 

- Dr. Anne Hillegas, ARA Corporation - Dr. Juan Vitali, ATEC 

- Mr. John Illgen, Northrop Grumman - Mr. Tom Wissink, Lockheed Martin 

- RADM Bert Johnston, USN (Ret), Wyle Labs - Mr. Bill Yeakel, ORSA Corporation 
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NDIA Test & Evaluation Division 


The Test & Evaluation Division was initially established under the auspices of the American Defense 
Preparedness Association (ADPA). With the merger of the National Security Industrial Association 
and the American Defense Preparedness Association on March 1, 1997, the T&E Division and its 
responsibilities were transferred to the National Defense Industrial Association. It is one of the 32 
divisions of the Association. 


NDIA TEST & EVALUATION DIVISION OBJECTIVES 

NDIA’s principal missions are to improve weapons technology, improve defense management, and 
maintain a strong science-industry-defense team continually responsive to all needs of the research, 
development, test & evaluation, production, logistics and management phase of national preparedness. 
It provides a forum for the exchange of ideas and information between its members and government 
agencies through a network of divisions, chapters, national and local meetings, and conferences and 
visits to the Department of Defense installations. Consistent with, and in furtherance of, the By-Laws 
and policies of the NDIA, the objectives of the T&E Division with respect to those government elements 
appropriate to its field of interest are: 


Ensure the continuation, responsiveness, and where necessary, the revitalization of the industrial base 
to support our national security by: 


Maintaining effective liaison with the Department of Defense, Executive and Legislative Branches 
of the Government, and other governmental departments and agencies to apprise appropriate 
representatives of these organizations of T&E Division activities and keep abreast of current and 
future developments. 


- Reviewing, evaluating and providing recommendations with regard to government policies, practices, 
directives and specifications including pending legislation concerning life cycle logistic support, supply 
chain integration, strategic and tactical mobility and business processes. 


- Serving as a catalyst for community dialogue among T&E communities within government, industry 
and academia, as well as provide an overarching means for dynamic interchanges of information with 
other functional areas. 


Sponsoring government/industry meetings, seminars and symposia to provide effective 
communications between government and industry relative to management activities for which the 
T&E Division is responsible under this matter. 


Maintaining viable liaison with other divisions of NDIA, other associations and national-level 
industry coalitions for the purpose of exchanging information of mutual interest and coordinating 
and cooperating on activities requiring joint participation. 


- Developing strategic directions effecting T&E partnerships between national security industries and 
government and to formulate proposals for national policies, addressing their impact on the national 
security industrial base. 


- Establishing and maintaining a productive proactive dialogue with the national news media. 
Fostering learning of the basics and advanced concepts and techniques of T&E within the 
defense industry including the cross-pollination of commercial approaches to T&E for the defense 


community. 


Commemorating and recognizing services rendered by individuals, companies, and government 
agencies in meritorious acts related to activities enumerated above. 


Monday, March 12, 2007 


10:00 AM - Conference Registration 
5:00 PM 
1:30 - 3:00 PM CONFERENCE TUTORIALS 


Tutorial Sessions are provided free of charge to all those registered for the 
Conference. Your attendance at these valuable tutorials is encouraged. 


TUTORIALS: Sessions Tl - T4 


Archer East Tutorial T1. Innovative Acquisition Methodologies in Support of Test & Evaluation 
Ballroom Objectives - Mr. Jeffrey Manthos, Program Analyst, GSA 


Archer West Tutorial T2. The Critical Link Between Test & Evaluation and Modeling & Simulation 
Ballroom - Dr. Mark Kiemele, President and Co-Founder, Air Academy Associates 


Danner East Tutorial T3. Reliability Growth Applied to Defense Issues - Dr. Larry Crow, President, 
Ballroom Crow Reliability Resources, Inc. 


Danner West Tutorial T4. DoD Acquisition Policy on Suitability KPP and Related Issues - Dr. Paul 
Ballroom Alfieri, Director of Research, DAU 


3:00 - 3:15 PM _ Tutorial Afternoon Break 


3:15 - 5:00 PM TUTORIALS: Sessions T5 - T8 (Continued Sessions) 


Archer East Tutorial T5. Innovative Acquisition Methodologies in Support of Test & Evaluation 
Ballroom Objectives - Mr. Jeffrey Manthos, Program Analyst, GSA 


Archer West Tutorial T6. The Critical Link Between Test & Evaluation and Modeling & Simulation 
Ballroom - Dr. Mark Kiemele, President and Co-Founder, Air Academy Associates 


Danner East Tutorial T7. Reliability Growth Applied to Defense Issues - Dr. Larry Crow, President, 
Ballroom Crow Reliability Resources, Inc. 


Danner West Tutorial T8. DoD Acquisition Policy on Suitability KPP and Related Issues - Dr. Paul 
Ballroom Alfieri, Director of Research, DAU 


5:00 - 6:30 PM __ Evening Reception/Display Area Open 


6:30 PM Display Area Closed for the Day 


Your attention please: Survey sheets will be given to every Conference registrant upon arrival. The survey 


sheets are intended to gather your thoughts and ideas regarding this NDIA T&E Conference. Registrants 
are requested to thoughtfully complete these surveys and return them to the NDIA desk prior to their 
departure. The results of these surveys will serve as a basis in planning for future conferences. 


; So SR 


% Soe 


The Westin Resort, Hilton Head Island, SC 
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Tuesday, March 13, 2007 


7:00 AM - 8:00 AM__ Continental Breakfast in the Display Area 


7:00 AM- 5:00 PM _ Conference Registration 


_ - Session A - 
Session Chair: Mr. James O’Bryon 
8:00 AM Call to Order and Remarks - Mr. Sam Campagna, Director, Operations, NDIA 
8:05 AM Tribute to Our Nation, National Anthem - Ms. Kera O’Bryon 
8:10 AM Welcome and Conference Introductory Remarks - Mr. James O’Bryon, Chairman, NDIA Test & 


Evaluation Division 


8:20 AM 
Conference Keynote Address: Assuring Quality and Sustainability in a Competitive 
Market - Mr. Keith E. Williams, President & CEO, Underwriters Laboratories 


9:00 AM 
Government Keynote Address: Getting Our Arms Around Motivating and Testing 
for Sustainability - Honorable Charles E. McQueary, Director, Operational 

Test & Evaluation, OSD 


9:40 AM Keynoters’ Dialogue - Mr. Keith E. Williams, President & CEO, Underwriters Laboratories and 
Honorable Charles E. McQueary, Director, Operational Test & Evaluation, OSD 
(**Audience may submit written questions to either speaker during this period.) 


10:05 AM Morning Break & Networking in the Display Area 


— Session B - 
Session Chair: Mr. James O’Bryon 


10:35 AM OTA’s Roundtable: Military Perspectives on Testing for Sustainability and Life Cycle Costing 
Chair: MG James Myles, USA, Commanding General ATEC 
Panelist: RDML William McCarthy, USN, Commander OPTEVFOR 
Maj Gen Robin Scott, USAF, Commander AFOTEC 
COL Michael Bohn, USMC, Commander MCOTEA 


12:15 PM MU rate at-Xe) a] 


- Session C - 
Session Chair: Dr. Mark Kiemele 


1:30 PM Statistically-Based Test Optimization Strategies - Dr. Neal Mackertich, Raytheon Integrated 
Defense Systems 


2:10 PM Report from the NDIA Industrial Committee on Operational Test & Evaluation (ICOTE) - Mr. 
Larry Graviss, Vice President, Jacobs Technology, Inc. 


2:20 PM Afternoon Break & Networking in the Display Area 


Tuesday, March 13, 2007 - Concurrent Focus Sessions 


2:50 - 5:30 PM 


Focus 
Session D1 


ARCHER EAST 


BALLROOM 


Session Chair: 
Dr. Anne 
Hillegas 


2:50 — 5:30 PM 


Focus 
Session D2 


BARNWELL 
BALLROOM 


Session Chair: 
Mr. Bill 
Yeakel 


2:50 - 5:30 PM 


Focus 
Session D3 


ARCHER 
WEST 


BALLROOM 


Session Chair: 
Mr. Dick 
Dickson 


5:30 PM 
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— Session D - 
Planning & Implementing Systems Engineering/T&E to Address 
Sustainability 
FS D1.1 T&E for Verifying Technology Development and Maturation - Mr. Chris 
DiPetto, Deputy Director for DT&E, OUSD (AT&L)/SSE 


FS D1.2 Early Application of Computer Program Systems Integration, Test and 
Performance Measurement - Mr. Thomas Sobieralski, CSC 


FS D1.3. Using Technology Readiness Assessments (TRA’s) to Assess the Maturity of 
Life Cycle Related Technologies - Dr. Jay Mandelbaum, IDA 


FS D1.4 Results of FY 2006 Test Resources Management Center Range Encroachment 
Survey Provided to the August 2006 TRMC Infrastructure Review - Mr. William Egan, 
IDA 


FS D1.5 Test and Training Enabling Architecture, Use to Prototype Joint Mission 
Environment Test Capability JMETC) - Mr. Richard Lockhart, Defense Test Resource 
Management Center 


FS D1.6 TRACER: A Tradespace Analysis Framework - Mr. Gerry Belcher, LMI 


Reducing Total Cost of Ownership & Realistic Estimates of Reliability 


FS D2.1 Low Cost Epoch-by-Epoch Network Centric Positioning Unit (ENPU) for FCS 
Testing - Dr. Jeffrey Fayman, Geodetics, Inc. and Dr. Lydia Bock, Geodetics, Inc. 


FS D2.2 Testing of the GPS SAASM End-to-End Functionality on Operational Weapon 
Systems Without the Availability of Signal in Space - Mr. Jim Killian, 46 Test Squadron, 
Holloman AFB 


FS D2.3 The Costs of Unsuitability and Benefits of Building in Reliability, Availability 
and Maintainability - Dr. Ernest Seglie, DOT&E 


FS D2.4 A Successfully Implemented Coordinated Subsystem Reliability Growth 
Planning Approach- Mr. Louis Chenard, Bombardier & Mr. Paul Ellner, AMSAA 


FS D2.5 Understanding the Combined Influence on Ownership Cost of Reliability, 
IW FeVoale-Uear-Vo)i tia vamm @Xo)aay ole) ai-d alam z-\el.¢-\el late pam @xolealaateyar-liinvam-valeim aol e-tcml a-1a re) gaal- ls -eeeme Dy 
James A. Forbes, LMI 


FS D2.6 Using Mission Essential Task Lists as a Basis for Mission-Based Operational 
Test Planning - Mr. James Carpenter, AVW Technologies 


Test Technology 


FS D3.1 Hazard Assessment of the SM-3 Block IA Missile — Mr. William Houchins, 
NSWC Dahlgren 


FS D3.2 The Four Element Framework: An Integrated Test & Evaluation Strategy - Mr. 
Christopher Wilcox, ATEC, APG 


FS D3.3 Operability Testing for High Availability Systems - Mr. Edward Beck, CSC 


FS D3.4 Demonstrating Operational Suitability Data Merging During Developmental 
Tests: M&S vs. Raw Data, What is Needed? - Mr. Jerome Perkins, SAIC 


FS D3.5 746 Test Squadron Testing of the Latest Embedded GPS/INS (EGI) Hybrid 
Navigation System for the F-16 Fighting Falcon - Mr. Jim Killian, 46 Test Squadron, 
Holloman AFB 


FS D3.6 Qualification Testing of the EFSS 120mm Mortar Ammunition - Mr. Patrick 
Freemyers, NSWC Dahlgren 


aves Ds Pw am ©) ol-1e-Va(o)ar-lmAVar-UNVATMo) mV (ole K-1g am @lolare|id(olamst-bX-lemm Fl laie-tar-Uale =m (O15), ce) 
Enhance System Sustainability - Dr. Don Gaver, NPS 


Afternoon Sessions Adjourn/Display Area Closed for the Day 


March 12 - 15, 2007 


Tuesday, March 13, 2007 - Evening Activities 


6:30 PM NDIA Honors Banquet 


Presentation of the Walter W. Hollis Award for Lifetime Achievement in Test & Evaluation 
This event is one of the highlights of our NDIA T&E Annual Conference. This year, we again take time 
to honor an individual who has demonstrated a lifetime commitment and contribution to the honest and 
realistic conduct of Test & Evaluation in support of the defense of the United States. This year, we are proud 
to honor such a person: Mr. James F. O’Bryon. 


Guest of Honor Mr. James F. O’Bryon 


Mr. James O’Bryon’s career began at IBM’s R&D Center in NY while still in 
undergraduate school. Subsequent to this, he took a position in the Actuarial 
Department of New York Life. The Army drafted him in December 1964 and this 
moved him into the T&E world. He held RDT&E positions at the Aberdeen Proving 
Ground from 1965-1986. In 1986, he accepted the position as the first Director, 
Live Fire Testing in the Pentagon and retired from the Pentagon, November 2001 as 
Dep. Dir. OT&E/LFT&E, a Deputy Assistant Secretary of Defense position. Jim has 
an undergraduate math degree and graduate degrees from GWU in Ops Research 
and from MIT in Electrical Engineering. 


Jim has chaired the NDIA T&E Division for 14 years and chaired dozens of national 

and international T&E Conferences. He has taught T&E at NDU, ICAF, NPS, University 

of Texas and other places for nearly 20 years and authored over 90 technical 

reports and 80 open literature articles on T&E and related issues. He has just 

completed an 800-page book on Lessons Learned from Live Fire Testing and has 
testified before Congress on several occasions on T&E and Aviation Security. He has received the ADPA 
Gold Medal, the NDIA Gold Medal, the Arthur Stein Award for Excellence in Live Fire Testing and numerous 
DoD Awards including two personally presented by two different Secretaries of Defense. He now chairs a 
Committee of the National Academy of Sciences, as well as serving as a consultant to several commercial 
and not-for-profit institutions. 


Previous Recipients 

2006: RADM Bert Johnston, USN (Ret), Wyle Laboratories : Mr. G. Thomas Castino, Underwriters Laboratories, Inc. 
2005: Hon Thomas Christie, DOT&E, OSD : Hon Philip E. Coyle, Ill, DOT&E, OSD 

2004: Dr. Marion L. Williams, Headquarters AFOTEC : Mr. Walter W. Hollis, Department of the Army 

2003: Mr. James W. Fasig, Aberdeen Test Center 


Guest Speaker Mr. Raymond Daddazio 


It is our special privilege to welcome as our guest speaker, 
Mr. Raymond Daddazio, President, Weidlinger Associates, 
New York, NY. September 11, 2001 is a day we will always 
remember, and the scenes of the World Trade Center 
collapsing is forever etched in our minds. 


Many were shocked not only by the aircraft attack 
against the buildings, but also see them collapse from 
the attack, resulting in a huge loss of life and property. 
Weidlinger Associates was called upon to lead the efforts 
to assess what and how the WTC collapsed and also how 
to mitigate a repeat of such a failure if future attacks 
occur on other major structures. Mr. Daddazio will share 
with us his insights into their analysis and 
recommendations. 


Note: Guests wishing to attend with registrants are welcome to attend this Banquet. However, additional tickets must be purchased 
at the NDIA desk prior to the Banquet. 


Wednesday, March 14, 2007 


AM- 


9:05 AM 


9:35 AM 


10:05 AM 


10:35 AM 


11:15 AM 


12:00 Noon 


Continental Breakfast in the Display Area 


Conference Registration 


- Session E - 
Session Chair: Mr. William Yeakel 


Call to Order and Remarks - Mr. Sam Campagna, Director, Operations, NDIA 


Warfighting in a Climate Warming World: The Expectations, the Options, and the 
Solutions - Honorable Philip E. Coyle, Ill, Former Director, OT&E, OSD 


A Concept for Suitability Assessment in Capabilities-Based T&E - Dr. Vincent P. 
Roske, Institute for Defense Analyses 


Reliability Growth as Applied to Military Systems - Dr. Larry Crow, President, 
Crow Reliability Resources, Inc. 


Overcoming the Lowest Bidder with Quality Products in Government Purchases - Mr. 
William Britz, President, Federal Program Management 


Morning Break & Networking in the Display Area 


_ — Session F -—_ 
Session Chair: Mr. Dick Dickson 


Development, Test & Evaluation of Unmanned Ground Vehicles/Robots, VADM 
Joseph Dyer, USN (Ret), President and General Manager, iRobot Corporation 


Industry Panel: 

Sustainability KPP: Industry Weighs in on Bidding for It and Building for It - Mr. John 
Stoddart, President, Defense Business, Oshkosh Truck 

Past, Present & Future Trends in RM&A - Mr. Ron Mutzelburg, Vice President, Boeing 
Phantom Works, DC 


Annual Awards Luncheon 


The Tester of the Year Awards are presented annually to civilian and military personnel supporting 
all of the military services and OSD. For each military service and OSD, an award is presented to the 
outstanding defense civilian, outstanding military, and outstanding supporting contractor. 


** Note: Guests wishing to attend with registrants are welcome to attend this event. However, additional tickets must be 
purchased at the NDIA desk prior to the luncheon. 


Tester of the Year Award Winners 


Military Civilian Contractor 

MAJ Michele Patello, USA Mr. Dana Blankenbiller Mr. Kenneth Van Allen 
Col Rich Stuckey, USAF Ms. Daria Stafford Ms. Crystal Villarreal 
Capt Peter Sandness, USAF Mr. William Pope Mr. Charles Reed 
LCDR Matthew Rising, USN Mr. Edward Lee Graham Mr. Jerry Manthei 

Maj James Roudebush, USMC Mr. Trevor Strand Mr. Todd Sweeney 


OSD Special Category 


ATEC Counterterrorism Test & Evaluation Team 
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1:15 PM 


1:45 PM 


3:15 PM 


- Session G - 
Session Chair: Mr. Jack Sheehan 


Coping with Reliability, Sustainability and Life Cycle Costs at DHS - Dr. George 
Zarur, Science Advisor, Science and Technology, DHS 


DT&E Panel: Systems Engineering and DT&E for System Sustainability 


Chair: Col Richard Stuckey, USAF, Principal Assistant Deputy Director for 
DT&E, OUSD (AT&L)/SSE 
Panelists: Dr. David Jerome, Deputy Director of Air, Space and Information 


Operations, AF Materiel Command 

Mr. Ray Lytle, Director, Life Cycle Engineering, Raytheon Missile 
Systems 

Mr. Brian Simmons, Director, US Army Evaluation Center, Army Test & 
Evaluation Command 

Mr. Richard Shubert, VP and Chief Engineering, Lockheed Martin 
Integrated Systems and Solutions 


Afternoon Break & Networking in the Display Area 


Wednesday, March 14, 2007 - Concurrent Focus Sessions 


3:30 - 5:30 PM 
Focus 
Session H4 


BARNWELL 
BALLROOM 


Session Chair: 
Dr. Lowell 
Tonnessen 


3:30 - 5:30 PM 


Focus 
Session H5 


ARCHER EAST 
BALLROOM 


Session Chair: 
Mr. Dick 
Dickson 


3:30 - 5:30 PM 


Focus 
Session H6 


ARCHER WEST 
BALLROOM 


Session Chair: 
Dr. Anne 
Hillegas 


5:30 PM 


— Session H - 
Test Methodology 


FS H4.1 Revisiting Quantitative Methods for Evaluating Training Programs for Systems 
Undergoing OT - Dr. Christopher Hekimian, SAIC 


FS H4.2 Producing Anywhere, Anytime Test, Evaluation and Diagnostics Capable 
Products to Eliminate the T&E Logistic Burden - Mr. Ryan Kinney, Agilent Technologies 


FS H4.3 Ship Suitability Testing - Preparing for the Future - Mr. Steve Fisher, NAVAIR, 
Patuxent River 


FS H4.4 Innovative Techniques to Solve Measurement and Detection Problems - Dr. 
Michael Slocum, Air Academy Associates 


Test Planning 
FS H5.1 A Continuum of Testing for Military Systems - Dr. Patricia Jacobs, NPS 


FS H5.2 CTEC (Collaborative Test & Evaluation Capability) and the Event Streaming 
Media System - Mr. Reid Johnson, NUWC Keyport 


FS H5.3 Integrated Weapons System Testing - Ms. Laura DeSimone, NSWC Dahlgren 


FS H5.4 Estimating Durations and Trials to Success in Test Trial Programs - Dr. Danny 
Hughes, LMI 


FS H5.5 Operational Testing of Procedures is Critical to Effective Use/Integration 
of Emerging Capabilities: Report on Joint Space Control Operations - Col Christian 
Daehnick, USAF, Negation Joint Test & Evaluation 

Technology to Reduce Life Cycle Cost 

FS H6.1 T&E of Electromagnetic Railguns - Mr. Thomas Boucher, NSWC Dahlgren 


FS H6.2 Cost Efficient Risk Management Through Integrated T&E Throughout the 
Systems Engineering Life-Cycle - Mr. Joseph Tribble, AVW Technologies 


FS H6.3 Sustainment of the Deployed Navy Munitions Inventory Through Continuous 
Quality Evaluation Test & Evaluation - Mr. Jeffrey Johnson, Navy Indianhead 


Display Area Closed for the Day 


Thursday, March 15, 2007 


Continental Breakfast in the Display Area 


Conference Registration 


_ — Session | - 
Session Chair: Mr. Jack Sheehan 


Call to Order and Remarks - Mr. Sam Campagna, Director, Operations, NDIA 
Suitability - At What Cost? - Dr. Paul Alfieri, DAU 


Service T&E Executives Speak 

Chair: Mr. Walt Hollis, Former DUSA/OR 

Panelists: Mr. Dave Duma, Principal Deputy, DOT&E 
Dr. Hank Dubin, Army T&E 
Mr. David Hamilton, Deputy Director, AF/TE 
Mr. Steven Whitehead, OPTEVFOR 


9:05 AM The Costs of Unsuitability and Benefits of Building in Reliability, Availability and 
Maintainability - Dr. Ernest Seglie, DOT&E 


9:30 AM Army T&E - Providing Essential Information to an Army at War - Mr. Brian Simmons, 
Army DTC 


10:00 AM Morning Break & Networking in the Display Area (Last Opportunity to Visit Displays) 


- Session J - 
Session Chair: Mr. William Yeakel 


10:20 AM Reliability-Based Design, Development and Sustainment - Dr. Anne Hillegas, Applied 
Research Associates 


11:15 AM Synthesis Panel”: Actions and Conference Takeaways 
Chair: Dr. Ernest Seglie, Science Advisor, DOT&E 
Panelists: Mr. Stephen Daly, DDOT&E, Conventional Systems 
Mr. Chris DiPetto, Deputy Director for DT&E, OUSD (AT&L)/SSE 
Ms. Diana Echols, Director of Customer Service Technology, 
Bell Helicopter Textron 
Mr. Steven Whitehead, Technical Director, OPTEVFOR 


“The discussion of this panel on the overall thrust of the Conference will 
serve as the basis of a White Paper and an overview article to be published 
in NDIA’s National Defense Magazine. 


11:55 AM Conference Closing Remarks - Mr. James O’Bryon, Chairman, NDIA Test & Evaluation 
Division 


Note: Please be sure to return your completed survey forms to the NDIA 
registration desk. 


“The Department of Defense finds this event meets the minimum regulatory standards for attendance by DoD employees. This finding does 
not constitute a blanket approval or endorsement for attendance. Individual DoD component commands or organizations are responsible for 
approving attendance of its DoD employees based on mission requirements and DoD regulations.“ 
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Promotional Partner 


Ant 


ACADEMY 
ASSOCIATES 


Simplify, Perfect, Innovate 
www.airacad.com 


Air Academy Associates has been a leader in providing continuous and breakthrough improvements 
to idustry since 1990. We help our clients generate significant Return on Investment by targeting all 
areas of the organization - development, design, manufacturing, and operations- and using sound 
business-enhancing methodologies to increase productivity, improve the efficiency of business 
operations, and provide customer satisfaction at a profit. 


Air Academy Associates’ Competitive Excellence Model provides our clients with the plan, strategy, 
and tactics - including the methods, tools, and techniques - needed to preserve market share, 
profitability, and brand recognition and to enable the client organization to evolve by developing 
new life cycle curves for new products/services or new generations of products/services. Our A3CE 
model, which has at its foundation our Knowledge Based Strategy, incorporates Lean, Six Sigma, and 
Systematic Tactical Innovation as Preservation activities and Design for Six Sigma and Systematic 
Strategic Innovation as Evolution activities. 


Air Academy Associates 
Competitive Excellence (A°CE) Model 


Te DesigNNOVATION™ 
Societal Needs 


Synchronizing the Business 
Aligning the Application of Methods to Strategy 
= 
2 
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ALes in 
Lean Six Sigma Driven by the Voice 
of tl tomer 


Knowledge-Based Strategy 
Voice of the Customer + Societal Needs —» Strategic Development 


© 2006 Air Academy Associates 


. 


Air Academy Associates differentiates itself and provides value to our customers by: 


Educating company leaders in our proven Knowledge-Based Business Strategy which emphasizes 
Voice of the Customer, Strategic Planning, and Systematic Innovation 
+ Customizing our support and materials based on the unique needs of the client 

Utilizing mature and experienced consultants who are proficient at problem-solving and who 
excel as classroom instructors and knowledge transfer agents 

Incorporating advanced methods and techniques such as Robust Design, Tolerance Allocation, 
Expected Value Analysis, Axiomatic Design, etc., into our programs to optimize current performance 
and to enhance design and development phases 
- Developing supplemental strategic alliances to insure that our clients’ needs are met and enabling 
clients to achieve quick self-sustainability 


For assistance in implementing a program of continuous innovation and overall competitive excellence, 
contact an Air Academy Associates Business Development representative at 1(800) 748-1277. 


See you next year at the 24th Annual 
National Test & Evaluation Conference! 


Hilton Palm Springs, CA - February 25 - 28, 2008 


=|] = 
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Test & Evaluation Conference 


“Test & Evaluation in Support of Operational Suitability, 
Effectiveness and Sustainment of Deployed Systems” 


ARE 


ACADEMY 
~ ASSOCIATES 


Simplify, Perfect, Innovate 


Thank You to Our 
Conference Promotional 
Partner! 


BA 
Suitability ...at what cost? 


“5-minute” warm-up act for the T&E Service Exec Panel 


Talk about 3 things: 
1. New “Materiel Availability” KPP 
2. DAU Suitability Research Project 
3. Announce NDIA/DAU TST-301 2007 All-Star Team ! 


THE JOUNT STAFF 
WASHINGTON, 0-0. 20373-8000 


JRCK OM 1616 
17 August 20h 


MEMORANIUM FOR: Under Secretary of Defense for Acquisition, Technology, 
and Logistica 
Commander, US Join Forces Commend 
Vice Chief of Sta, Us ATMY 
Vice Chief of Naval Operations 
Vice Chie! of Staif, US Air Force 
Assistant Commandant of the Marine Corps 


Subject; Key Performance Parameter Study Recommendations ang 
Iniplemencation 


i, The Joint Requirements Oversight Couticil (JROC) approved che Key 
Perlormance Parameter (KPT) Study recommendations, The JROC endorses 
the implementation of 4 mandared Matericl Avadiability EPP with supourting 
key system attoibutcs of matericl reliability and ownershia cost for all Major 
Nefense Acquisition Prograros (MDAPs) and select ACAT II ard IIT programs. 
The IRGC also endorsed selectively applying an Encrey Efficiency RPP and a 
System Training KPP, as appropriate. 


2. Ty better ensure the correct KPPs are selected, the JRO encoracd the se 
af SPP reference sheets produced as part of this study, The KPP referericc 
sheets will be used as an aid in the procesy of idertifying ard validating 
potential KPPa for any acquisition program. - 


3. Implementation of the study recommendations wil] be curicucrent with rhe 
publishing of the nest revision of CICS 3170-series documents. ‘The revision 
will incerporate the details of the execution and will be coardinated for final 
release by 341 October 2006. Specific FROG implementation due backs and 
approved recommendations ace enclosed, 


Admiral, US Navy 
Vie Chairmen 
of the Jame Chicts of SialF 
Enctosure 


JROC Memo: 17 Aug 2006 


(Subj: Key Performance Parameters Study Recommendations and Implementation) 


. Endorsed Mandatory “MATERIEL AVAILABILITY” Key Performance 
Parameter (KPP) for all MDAPs and Select ACAT Il and Ill 


With 2 Supporting Key System Attributes (KSAs): 
A. Materiel Reliability KSA 
B. Ownership Costs KSA 


. Endorsed ENERGY EFFICIENCY KPP for selected programs, as 
appropriate 


. Endorsed TRAINING KPP for selected programs, as appropriate 


. Did not endorse requirement for mandatory KPPs for these criteria: 
COST 


TIME and/or SCHEDULE 

SUSTAINMENT 

COALITION INTEROPERABILITY 

FORCE PROTECTION AND SURVIVABILITY : 


JROC Approved* Mandatory Sustainment KPP and KSAs 


e Single KPP: 
; 7 TP _ Number of End Items Operational 
* Materiel Availability ( Total Population of End Items ) 


e Mandatory KSAs: . 
- Materiel Reliability (MTBF)(= _2t@! Operating Hours ) 


Total Number of Failures 
¢ Ownership Cost (O&S costs associated w/materiel readiness) 


¢ For mission success, Combatant Commanders need: 
e Correct number of operational end items capable of performing 
the mission when needed 
e Confidence that systems will perform the mission and return 
home safely without failure 
¢ Ownership Cost provides balance; solutions cannot 


be availability and reliability “at any cost.” 


*JROC Approval Letter JROCM 161-06 Signed 17 Aug 06; 


Revised CJCS 3170 will put into Policy 
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LCC Distribution 
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BA) Life Cycle Management 


Sustain The 
Design 


Design For 
Sustainment 


65-80% of the Life Cycle Cost 


Concept 


Concept 
Decision 


Pre-Systems 
Acquisition Systems Acquisition Sustainment 


USD(AT&L) FY 07 Strategic Goals (#4) Emphasize Sustainment Outcomes 
Throughout The Life Cycle Management Process 


——_—_—_—_—_——<_—__—<—o 44 yis 
—_—_—_—_—_—_———>_ S6yis 
51 yrs 
36 yrs 


1940 1950 1960 1970 1980 1990 2000 2010 2020 2030 2040 


SOURCE: John F. Phillips DUSD (L) 


Life Cycle Costing Considerations 


“As Government expenditures, those 
due to broken down chariots, worn- 
out horses, armor and helmets, 
arrows, and crossbows, lances, hand 
and body shields, draft animals and 
supply wagons will amount to 60% 
of the total.” 


Sun Tzu (The Art of War, 6 Century B.C.) 
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Suitability ...at what cost? 


DAU Research Project: 
“STRYKER Suitability Analysis” 


Dr. Paul Alfieri, 

Director of Research 

Defense Acquisition University 
paul.alfieri@dau.mil 

(703) 805-5282 


Dr. Don McKeon, 

Professor of Engineering Management 
Defense Acquisition University 
mckeond@tacom.army.mil 

(586) 574-7240 


Suitability ...at what cost? 


Typical IOT&E Evaluation Results: 
EFFECTIVENESS: approximately 90% success rate 
SUITABILITY: approximately 60 - 75% success rate 


Typical Decision after IOT&E: Begin fielding ASAP, even before... . 
Suitability problems are addressed 
Reliability is improved 
Maintenance procedures are mature 
Training is complete 


Why field before addressing these problems? Urgent Combat Need 


The QUESTION: How much does it cost us to do business this way? 
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Suitability ... at what cost? 


DAU Research Study Proposal 
Investigate various types of systems 


Total of 5 or 6, several from each service 
Criteria: 

Recently fielded 

Evaluated to be Effective but not “fully” Suitable 
Examine performance of systems wrt suitability 
Determine suitability cost drivers 
Evaluate suitability trends 


Sponsor Decision: Start with one program, work from there..... 


First Program Selected: STRYKER Family of Vehicles 


Additional Study Candidates: TBD 
alg 


” STRYKER FAMILY OF VEHICLES 
B= In service with the US Army 


A B C 


Legend 

A. Infantry Carrer Vehicle B. Command Vehicle C. Mobile Gun System 
D. Fire Support Vehicle  E. Medical Evacuation F. Mortar Carner 

G. Engineer Squad Vehicle H. Anti-tank Guided Missle |. NBC Reconnaissance 


J. Reconnaissance Vehicle 
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Results to date: 


— Analysis of CDRL data ongoing 
— Established process and methodology 
— Developed parametric models 


— GOAL: independent CPM determination 


Cost Per Mile (CPM) Estimates 


¢ CPM estimate - $17.19 (GAO 04-925, including labor, parts & repair) 
¢ CPM estimate - $18.78 (Stryker R-TOC Brief) 


¢ CPM estimate - $18.23 (based on M113 methodology w/Stryker 
adjustments) 


¢ CPM estimate - $14.53 (based on initial 4 month deployment data) 


Current: 
¢ CPM estimate (GDLS) - $13.52 garrison 
$ 8.88 deployed 


¢ DAU CPM estimate- $13.30 garrison 
$ 7.95 deployed 


Recommendations 


Continue Research 

— Complete Stryker analysis 
Feedback from sponsor 
Feedback from community 
Determine path ahead 


Develop methodology for conducting 
suitability studies on other systems 


Look at other programs for comparison 
— Other services, other types of systems 
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NDIA/DAU TST-301 2007 All Stars 


1B - Mr. Steve Whitehead, (COMOPTEVFOR) 
2B - Col Mike Bohn, USMC (MCOTEA) 

3B —- Joe Wascavage, (NAVAIR) 

SS - Brian Simmons, (ATEC) 

LF — Rick Lockhart, (DTRMC) 

CF — Dave Duma, (DOT&E) 

RF -— Steve Zink, (OSHKOSH TRUCKS) 

C -Dr. Ernest Seglie, (DOT&E) 

P - Jim O’Bryon, CIVLANT (NDIA) 


Pinch Hitters: COL Sam Kyle, USAF (AFOTEC) 
CAPT Rick Scudder, USN (formerly OP-091) 
Mr. Pete Nolte, (AT&L) 
Mr. Larry Leiby, (TEMA) 
Mr. Fred Myers, (AT&L) 
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BA 
Suitability ...at what cost? 


Dr. Paul Alfieri, 

Director of Research 

Defense Acquisition University 
paul.alfieri@dau.mil 

(703) 805-5282 


Dr. Don McKeon, 

Professor of Engineering Management 
Defense Acquisition University 
mckeond@tacom.army.mil 

(586) 574-7240 


Suitability ...at what cost? 


Typical IOT&E Evaluation Results: 
EFFECTIVENESS: approximately 90% success rate 
SUITABILITY: approximately 60 - 75% success rate 


Typical Decision after IOT&E: Begin fielding ASAP, even before... . 
Suitability problems are addressed 
Reliability is improved 
Maintenance procedures are mature 
Training is complete 


Why field before addressing these problems? Urgent Combat Need 


The QUESTION: How much does it cost us to do business this way? 
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Suitability ... at what cost? 


DAU Research Study Proposal 
Investigate various types of systems 


Total of 5 or 6, several from each service 
Criteria: 

Recently fielded 

Evaluated to be Effective but not “fully” Suitable 
Examine performance of systems wrt suitability 
Determine suitability cost drivers 
Evaluate suitability trends 


Sponsor Decision: Start with one program, work from there..... 


First Program Selected: STRYKER Family of Vehicles 
Additional Study Candidates: TBD 


” STRYKER FAMILY OF VEHICLES 
B= In service with the US Army 


A B C 


Legend 

A. Infantry Carrer Vehicle B. Command Vehicle C. Mobile Gun System 
D. Fire Support Vehicle  E. Medical Evacuation F. Mortar Carner 

G. Engineer Squad Vehicle H. Anti-tank Guided Missle |. NBC Reconnaissance 


J. Reconnaissance Vehicle 


Now, back to Suitability .... 
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Life Cycle Management 


Pre-Systems 
Acquisition Systems Acquisition Sustainment 


Life Cycle Costing Considerations 


“As Government expenditures, those 
due to broken down chariots, worn- 
out horses, armor and helmets, 
arrows, and crossbows, lances, hand 
and body shields, draft animals and 
supply wagons will amount to 60% 
of the total.” 


Sun Tzu (The Art of War, 6 Century B.C.) 


1940 1950 1960 1970 1980 1990 2000 2010 2020 2030 2040 
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SOURCE: John F. Phillips DUSD (L) 


DoD Directive 


(5000.1 


“PMs shall consider supportability, 
life cycle costs, performance, and 
schedule comparable in making 


program decisions.” 


Department of Defense 


DIRECTIVE 


NUMBER 5000.1 
May 12, 2003 


USD(AT&L) 
SUBJECT: The Defense Acquisition System 


References: 


(a) DoD Directive 5000.1, “The Defense Acquisition System,” October 23, 2000 (hereby 
canceled) 

(b) DoD Instruction 5000.2, “Operation of the Defense Acquisition System,” May 12, 2003 

(c) DoD 5025.1-M, “DoD Directives System Procedures,” current edition 

(d) Title 10, United States Code, “Armed Forces” 

(e) Section 2350a of title 10, United States Code, “Cooperative Research and Development 
Projects: Allied Countries” 

(f) Section 2751 of title 22, United States Code, “Need for international defense cooperation 
and military export controls; Presidential waiver; report to Congress; arms sales policy” 

(g) Section 2531 of title 10, United States Code, “Defense memoranda of understanding and 
related agreements” 

(h) Federal Acquisition Regulation (FAR), current edition 

(i) Section 1004, Public Law 107-314, “Bob Stump National Defense Authorization Act for 
Fiscal Year 2003,” “Development and Implementation of Financial Management Enterprise 
Architecture” 

(j) DoD Directive 8500.1, “Information Assurance (IA),” October 24, 2002 

(k) DoD Directive 4630.5, “Interoperability and Supportability of Information Technology (IT) 
and National Security Systems (NSS),” January 11, 2002 

(I) DoD Directive 2060.1, “Implementation of, and Compliance with, Arms Control 
Agreements,” January 9, 2001 


1, PURPOSE 
This Directive: 
1.1. Reissues reference (a) and authorizes publication of reference (b). 
1.2. Along with reference (b), provides management principles and mandatory policies 
and procedures for managing all acquisition programs. 
2. APPLICABILITY AND SCOPE 


2.1. This Directive applies to the Office of the Secretary of Defense, the Military 
Departments, the Chairman of the Joint Chiefs of Staff, the Combatant Commands, the Office of 
the Inspector General of the Department of Defense, the Defense Agencies, the DoD Field 
Activities, and all organizational entities within the Department of Defense (hereafter 
collectively referred to as "the DoD Components"). 


2.2. The policies in this Directive apply to all acquisition programs. 
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AT&L Memo: 22 Nov 2004 


(Subj: Total Life Cycle Systems Management (TLCSM) Metrics) 


Emphasizes use of PBL (Performance-Based 
Logistics) for all weapons 


Provides Specific Definitions (and Formulas) for 
the following metrics: 
1. Ao (Operational Availability) 
2. Mission Reliability 
3. TLCS Cost per Unit of Usage 
4. Cost per Unit of Usage 
5. Logistics Footprint 
6. Logistics Response Time 
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THE UNGER SECRET4ARY GF DEFENSE 


3201¢ DEFEMBE-PENTARDN 
HABHINGTGN, AC 20901-8010 


NOW 5 3 D605 


400 Leatnee 
MEMCRANDUM FOR SECRETARIES OF TER MILITARY DEPARTMENTS 
(ATTN: SERVICE ACQUISITION EXECUTIVES) 


‘ SUSTSOT: Total Life Cvote Systems Management (TCCSM? Metrics 


The Defenye Besiness Bourd cevarsenanded to the Deputy Scarciary uf Defense 
thay the Peparunent aggressively porate implsmentiion of Ferfomanre-Beavee. 
Logistics, for ali its weapons, new and legacy, 


in a memorandum tated August 16, 2004, my predecessor dizooted mensuring 
performence in teams of Oprniiunal Availability, Mise or. Reliability, Cost per Unit of 
Usage, Logistics Footerint, and Logistirg Responte Um. For consistency, this 
memorandum provides specific definitions of those mejrics for uss across the Departnenr 
(attached), I direct their use as the standard set of metrics for cvalusting cverail TLOSM. 


I aise direct the TLCSM 2xecutive Council to develop a “TLCSM Matries 
Hendbook,” with specific metrics, formulas and cateutation methodologies. It willbe 
used in performance-based contracts and for sustainercat oversight. The handbook wilt 

" also define supporting daia tequiremonts thet should belincozporated inco emerging 
logistics Infortuatian systsuis. 


The principe! polnt of contuct for adinlaisiration of the handogok ts 
Mr. Lou Kratz, Assistant Deputy Under Secretary of Defense ae ea Plas 


anc Programs), 703-614-6327, Luni Keiz@osd. anit 


Agtashstient: 
As cated 
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JROC Memo: 17 Aug 2006 


(Subj: Key Performance Parameters Study Recommendations and Implementation) 


. Endorsed Mandatory “MATERIEL AVAILABILITY” Key Performance 
Parameter (KPP) for all MDAPs and Select ACAT Il and Ill 


With 2 Supporting Key System Attributes (KSAs): 
A. Materiel Reliability KSA 
B. Ownership Costs KSA 


. Endorsed ENERGY EFFICIENCY KPP for selected programs, as 
appropriate 


. Endorsed TRAINING KPP for selected programs, as appropriate 


. Did not endorse requirement for mandatory KPPs for these criteria: 
COST 


TIME and/or SCHEDULE 

SUSTAINMENT 

COALITION INTEROPERABILITY 

FORCE PROTECTION AND SURVIVABILITY . 


THE JOUNT STAFF 
WASHINGTON, 0-0. 20373-8000 


JRCK OM 1616 
17 August 20h 


MEMORANIUM FOR: Under Secretary of Defense for Acquisition, Technology, 
and Logistica 
Commander, US Join Forces Commend 
Vice Chief of Sta, Us ATMY 
Vice Chief of Naval Operations 
Vice Chie! of Staif, US Air Force 
Assistant Commandant of the Marine Corps 


Subject; Key Performance Parameter Study Recommendations ang 
Iniplemencation 


i, The Joint Requirements Oversight Couticil (JROC) approved che Key 
Perlormance Parameter (KPT) Study recommendations, The JROC endorses 
the implementation of 4 mandared Matericl Avadiability EPP with supourting 
key system attoibutcs of matericl reliability and ownershia cost for all Major 
Nefense Acquisition Prograros (MDAPs) and select ACAT II ard IIT programs. 
The IRGC also endorsed selectively applying an Encrey Efficiency RPP and a 
System Training KPP, as appropriate. 


2. Ty better ensure the correct KPPs are selected, the JRO encoracd the se 
af SPP reference sheets produced as part of this study, The KPP referericc 
sheets will be used as an aid in the procesy of idertifying ard validating 
potential KPPa for any acquisition program. - 


3. Implementation of the study recommendations wil] be curicucrent with rhe 
publishing of the nest revision of CICS 3170-series documents. ‘The revision 
will incerporate the details of the execution and will be coardinated for final 
release by 341 October 2006. Specific FROG implementation due backs and 
approved recommendations ace enclosed, 


Admiral, US Navy 
Viee Chairmen 15 
of the Jame Chicts of SialF 
Enctosure 


JROC Approved* Mandatory Sustainment KPP and KSAs 


e Single KPP: 
; 7 TP _ Number of End Items Operational 
* Materiel Availability ( Total Population of End Items ) 


e Mandatory KSAs: . 
- Materiel Reliability (MTBF)(= _2t@! Operating Hours ) 


Total Number of Failures 
¢ Ownership Cost (O&S costs associated w/materiel readiness) 


¢ For mission success, Combatant Commanders need: 
e Correct number of operational end items capable of performing 
the mission when needed 
e Confidence that systems will perform the mission and return 
home safely without failure 
¢ Ownership Cost provides balance; solutions cannot 


be availability and reliability “at any cost.” 


*JROC Approval Letter JROCM 161-06 Signed 17 Aug 06; 


Revised CJCS 3170 will put into Policy 


Proposed Life Cycle Sustainment 
Outcome Metrics 


- Materiel Availability (KPP*) 
— A Key Data Element Used In Maintenance 
And Logistics Planning 
¢ Materiel Reliability (KSA*) 
— Provides A Measure Of How Often The System Fails/Requires Maintenance 
— Another Key Data Element In Forecasting Maintenance/Logistics Needs 
¢ Ownership Cost (KSA*) 
— Focused On The Sustainment Aspects Of The System 
— An Essential Metric For Sustainment Planning And Execution 
— Useful For Trend Analyses — Supports Design Improvements/Modifications 
¢ Mean Downtime 
— A Measure Of How Long A System Will Be Unavailable After A Failure 
— Another Key Piece Used In The Maintenance/Logistics Planning Process 


¢ Other Sustainment Outcome Metrics May Be Critical To Specific Systems, 
And Should Be Added As Appropriate 


* Sustainment KPP & KSAs Included In Revised Draft CJICSM 3170 


These 4 Life Cycle Sustainment Outcome Metrics Are Universal 


Across All Programs And Are Essential To Effective Sustainment Planning 


DUSD AT&L Metrics Evolution 


Life Cycle Sustainment Metrics 


TLCSM Metrics (Nov 05) (Feb 07) 


a mp 
a a 
ae =» 
mp 


(per Aug 06 JROC Memo) 


DAU Stryker Suitability Study 


e Interim Progress Report #2 
— Objectives 
— Process 
— Progress & Plans 
— Findings & Observations 
— Data Analysis 
— Reliability Measurement Issue 
— Challenges 
— Recommendations 
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DAU Stryker Suitability Study 


¢ Objectives 


— To conduct a research study to quantify the 
difference between projected O&S 
(associated with the RAM requirement) 
and the actual costs associated with the 
achieved level of operational suitability. 
That is, quantify the costs of not achieving 
adequate levels of operational suitability. 


Process 


e Phase 1- Initial Program (Stryker) 
a. Understand the problem 
b. Define detailed study objectives 
c. Collect data 
d. Analyze data and build models 
e. IPR at T&E Conference - Hilton Head 
f. Acquire additional data as needed 
g. Draft report 
h. Finalize report 


e Phase 2 - Analysis of 5 additional 
programs covering multiple types 
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Data Collection: 


Phase 1 Sources 
— Stryker PM Team (TACOM Warren, MI) 


— AEC RAM Directorate (APG) 
— OTC Reps (Ft. Hood) 

— AT&L Rep 

— IDA 

— LMI 

— GDLS CDRL Data 

— Ft. Lewis Stryker Team 
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Findings & Observations 


Warfighters very satisfied with Stryker 
performance in-theatre 


Brigade Commanders extremely happy with ICLS 


High Operational Readiness Rates, but ORR Is 
prioritized over support costs 


Op Temp in-theatre far exceeds planned usage 
rates (X10, X15, X30 7?) 


Operational Environment much different than 
expected 


Combat configurations add excessive weight to 
vehicle (affecting reliability and performance) 


Army did not buy Tech Data Pkg - “Prohibitively 
expensive” ... risk to government 
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Findings & Observations 


¢ Operational Readiness Rate not necessarily 
consistent with traditional Ao (Operational 
Availability) 
— RAM issues can be masked by ORR 


e Mission Completion vs. Subsystem Failure 


— Possibly leads to overestimating system reliability due 
to non-reporting on individual subsystem (component) 
failures 


— Multi-mission vehicle — with subsystem failures, system 
can still perform alternate missions 


¢ Reporting Criteria Issue: 
— ORR vs. MTBF of individual subsystems 
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Reliability Issues 


¢ Reliability requirement as defined in 
ORD 


— 4.3.1.3. The Stryker (vehicle only, excluding GFE 
components/systems) will have a reliability of 1000 mean 


miles between critical failure (i.e., system aborts). 
¢ Reliability issues and cost drivers found 
during DT/OT correlate well with fielded 


experience 


25 


Operational Environment 


e Field usage much harsher than planned 
— e.g., higher tire pressure, roads, curbs, weight (armor, sandbags) 
e Mission Profile says 80% XCountry, 20% Primary Roads 


— in-theater mission just the opposite ... most missions in urban 
environment (police action) on paved roads 


¢ OpTempo very high (>10X) 
— High OpTempo may improve reliability numbers, but beats up equipment 


— With low usage, seals can dry up, humidity can build up in electrical 
components 


e Changes in mission & configuration are putting excess stress on 
vehicle: armor/sandbags, over inflated tires, going over curbs 
— replacing 9 tires/day (>3200 tires/yr) 
— wheel spindles developing fatigue cracks 
drive shafts breaking 
— prescribed tire pressure is 80 PSI, however, with slat 
armor/sandbags — must maintain >95 PSI 
95 PSI is a logistics burden on operators 
e Must be maintained by the soldier (tire inflation system can’t do it) 


¢ Soldiers must check tire pressure more than 3 times per day to 2° 
maintain 95 PSI 


Tactical Considerations 


e Slat Armor design (additional 5000 Ib) is effective for many RPG 
threats, but negatively impacts circumference, weight and 
performance of Stryker 


— Causes multiple problems for safe and effective operation 


Slat armor on rear ramp too heavy - greatly strains lifting equipment 
— Occasionally, crews must assist raising/lowering ramp 
Bolts on rear ramp break off frequently with normal use 


Slat armor bends with continued ops ...can cover escape hatches and 
block rear troop door in ramp 


Slat armor interferes with driver’s vision 


Slat armor difficult for other traffic to see at night... Safety hazard in 
urban environment 


Slat Armor prohibits normal use of exterior storage racks 


— Significantly impacts handling/performance in wet conditions 


Adds excessive strain on engine, drive shafts, differentials 


— Impairs off-road ops 
¢ Though not designed primarily for the urban fight (MOUT), 
Stryker is well-suited for it 
— Unlike M-1, Stryker is “ghostly” quiet ... tactical advantage 


e Stryker overall OIF performance significantly better than 
HUMVEE, BRADLEY or M-1 in this environment 27 


Stryker Fleet Readiness 


Number Fielded 


ORR vs Strykers Fielded 
As Of: 20 Feb 2007 


100% 


Wi 98% 


+ 96% 


H+ Y4% 


92% 


90% 


+ 88% 
+ 86% 
84% 


PSST IOI GCI MIT 


Ga Strykers Fielded —#e— ORR 


ORR Percentage 


Operational Readiness Rate (ORR) 


¢ Contractual requirement: ORR > 90% 
— Does not include GFE (base vehicle configuration 
only) 
¢ Stryker consistently above requirement 
— Current ORR 97% (20 Feb 07) 
¢ Cost-plus-fixed-fee contract motivates GDLS 
to meetORR.... 


— However, contract does not incentivise controlling 
costs... risk to government 


— Example - to repair cracked hyd res in power pack, 
whole power pack is replaced in field 
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Cost Per Mile (CPM) 


CPM is a planning tool used to project future budget 
requirements 

No specific value of CPM required by contract 

Govt/Kr both calculate CPM independently, and use 
results to negotiate parts cost forecasts to determine 
purchasing requirements 

For this research project, DAU is doing our own 
independent computation of CPM (garrison and deployed 
units) to validate other data and our methodology 
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Cost Per Mile (CPM) Estimates 


CPM estimate - $17.19 (GAO 04-925, including labor, parts & repair) 

CPM estimate - $18.78 (Stryker R-TOC Brief) 

CPM estimate - $18.23 (based on M113 methodology w/Stryker adjustments) 
CPM estimate - $14.53 (based on initial 4 month deployment data) 


CPM estimate (GDLS) - $13.52 garrison 
$ 8.88 deployed 
DAU CPM estimate - $13.30 garrison 
$ 7.95 deployed 


Note 1 - We need to understand the basis for these estimates more 
thoroughly (assumptions, models, configurations, limitations .. . ) 


Note 2 - Figures above are averages across all variants (deployed or 
garrison) 


Note 3 - CPM higher for garrison than deployed stryker ??? 


Why? A. While deployed, non-essential maintenance can be delayed until 
absolutely necessary ... intervals between reported failures increases, 
CPM decreases 


B. Maintenance more accessible/available in garrison — follow the 
book closer 


C. Higher mi/day deployed... less labor/mi 31 


Other Findings... cont. 


¢ Stryker initial deployment/fielding was 
extremely accelerated to meet urgent combat 
need 
— Result was that Army was doing these things 
concurrently: 
¢ Testing 
e Producing 
¢ Fielding 
¢ Conducting combat operations 


¢ The threat and the operational environment 
were different than anticipated 
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Other Findings... cont. 


¢ Immature Maintenance Procedures- many procedures 
have not been validated in IETMs (interactive 
electronic tech manuals) lead to: 
— “Tribal System Maintenance” from experienced crews ( 


“|, . that new book isn’t any good........... this is the way 
it worked on the M113, so do it like this”) 


¢ With Kr support to maintain vehicles, soldier crews 
develop “rental car mentality” .. . 
— Lack of ownership mentality ... overly dependent on 
contractor 
— Sometimes they forget the basics (oil check) 
— One vehicle lost because pre-mission checks were ignored 
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DATA ANALYSIS 


Phase 1 — March 2007 


Data Collected 


¢ CDRL A003 (Aug 2006) 
— Parts Consumption Report (for ~ 1 yr) 


— Good quality data (possibly some errors in mileage or 
dates) 


¢ CDRL A004 (Aug 2006) 
— Repairable Items Repair Cost Summary 
— Most repair items have estimates or quotes 
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Cost Per Mile Analysis 


Labor + Replacement Parts 
Cost Per Mile = 


Total Vehicle Mileage 


Labor : $4.73M per brigade (average value) 
Replacement Parts : from CDRL A003 Consumption Report 
Part Repair : No historical data for many parts 
Variability in Part Repair 
Existing data from CDRL A004 
(Repairable Items Repair Cost Summary 
Vehicle Mileage : Does not exist for all vehicles 


Questionable accuracy : 


Determining the Average Repair Cost 


e Repair Cost data only exists for ~ 26% of total 
consumable parts 


— Determine the Average Repair Costs for Repairable 
Parts listed in CDRL A0OO4* 


— Determine Average Scrap Rates for Repairable Parts 
listed in CDRL A004 


e For remaining consumables (~74%): 


— Use Parametric Models developed from CDRL A004 
data 


* CDRL A004 Repairable Items Repair Cost Summary) ot 


Repair Costs Parametric Model 


Data from CDRL A004 

Uncertainty for parts or assemblies costing more than $50k 
Repair of Powerpack set to 30% 

Did not factor in warranty items 


100 @ Repair Cost (%) Without Scrap 
Model 


9 


80% 
70% 
60% 
50% 
40% 


Repair Cost (%) 


30% 

20% 

10% 

0% 

$0 $50,000 $100,000 $150,000 $200,000 $250,000 $300,000 
Unit Cost 


Scrap Rate Parametric Model 


- Model Used For Parts not in CDRL A004 
- High statistical variance for some parts due to small sample size 
- 100% data points ignored in the model 


120.0% 


100.0% 


80.0% 


60.0% 


Scrap Percentage 


40.0% 


20.0% -heg 


$0 $25,000 $50,000 $75,000 $100,000 $125,000 
Unit Cost 


Total Vehicle Mileage 


Labor + Replacement Parts 
Cost Per Mile = 


Total Vehicle Mileage 


Vehicle Mileage : Does not exist for all vehicles 
Questionable accuracy 


e Extreme values discarded. 
e Miles/day calculated for every vehicle in the database 


e Average miles/day from the database assumed to apply 
to all Brigade vehicles 
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Estimating Miles Per Day 


Earliest Part Latest Part 
Consumption Consumption 
(date & mileage) (date & mileage) 
For each 
vehicle 


Vehicle miles per day ~ (Miles, - Miles-)/(Day, - Day,) 


¢ Miles/day computed for each vehicle 
¢ Downtime not factored into the estimation. 


41 


Vehicle Miles Per Day From A003 


(CONUS) 


Miles/day 


50 


Maximum Limit For CONUS 


100 150 


200 250 
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Estimating the Repair Cost per Mile 


For each A 


ve h | cl € Earliest Part Latest Part 
Consumption Consumption 
(mileage) (mileage) 


e Above computation over estimates cost/mile because it 
doesn't include any mileage before the first or after the 
last part consumption 


e The error is a function of the number of failures 
(i.e., as the failures increase, the error decreases) 


¢ Numerical simulations were performed to develop a 
correction factor to be applied to the computed repair 


costs per mile b 


Correction Factor for the Estimated 


Repair Cost Per Mile 


CPM Correction Factor 


y = -3E-05x" + 0.0014x° - 0.0271x° + 0.2383x + 0.0918 
R* = 0.9971 


5 10 15 
Average Number of Failures 


20 


25 
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CONUS Cost Per Mile 


¢ CPM based on vehicles with: 
— Maximum total miles < 5,000 
— Maximum Miles/Day < 100 


¢ Models: 
— Parametric Repair Cost Model 
— Parametric Scrap Rate Model 
— Cost per Mile Correction 


¢ Assumptions 
— 300 Strykers Per Brigade (all operational) 
— Power Pack repair = 30% unit cost 
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CONUS Cost/Mile 


ICLS Labor, Replacement Parts, Part Repair 


Repair Cost | in Total Mileage | in | Spares/ Repair Miles Per 

vehicle Type | venicies | Computation | “Computation [Pars Costmte| "Day | Tot Cem 
$1,581,641 218,138 
$279,921 22,504 
$172,499 20,200 
$395,797 28,970 
$165,540 18,558 
$66,682 17,405 
$559,520 110,313 


$3,221,599 436,088 


Assumptions: Each vehicle < 5k total miles, < 100 miles/day 
average, 30% repair cost for Power Pack 46 


Deployed Cost Per Mile 


¢ CPM based on vehicles with: 
— Maximum total miles < 20,000 
— Maximum Miles/Day < 400 


¢ Models: 
— Parametric Repair Cost Model 
— Parametric Scrap Rate Model 
— Cost per Mile Correction 


¢ Assumptions 
— 300 Strykers Per Brigade (all operational) 
— Power Pack repair = 30% unit cost 
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Deployed Cost/Mile 


ICLS Labor, Replacement Parts, Part Repair 


No. Repair Cost | in Total Mileage | in | Spares/ Repair Miles Per 
$8,225,102 1,108,756 
$765,983 120,708 
$1,393,062 218,260 
$587,658 134,119 
$486,028 95,890 
$223,414 79,945 
$2,303,741 317,632 


All vehicles $13,984,989 2,075,310 


— Model assumes $4.73M per brigade 


— Higher miles/day for Deployed vehicles results in lower Total Cost Per 
Mile 


Assumptions: Each vehicle < 20k total miles, < 400 miles/day 
average, 30% repair cost for Power Pack 48 


Sensitivity Analysis 
(CONUS) 


e Using an Overall Average Repair Cost (based 
on CDRL AO04) Instead of the Parametric 
Models drops the CPM by 2% 


e Increasing the limit on Miles Per Day (from 100 
to 300) drops the CPM by 3% 

e Increasing the limit on Maximum Miles (from 
5,000 to 10,000) drops the CPM by 4% 
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Challenges 


Validity of comparisons 
Baseline assumptions 
Missing Data 

Quality of data 


Recommendations 


Continue Research 

— Complete Stryker analysis 
Feedback from sponsor 
Feedback from community 
Determine path ahead 


Develop methodology for conducting 
suitability studies on other systems 


Look at other programs for comparison 
— Other services, other types of systems 
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Edward Beck 
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EXPERIENCE. RESULTS. 


Agenda 


¢ Background 


e System Management Overview 


e Case Study: Insight 


e The Benefits of Insight 


3/16/2007 1:26:30 PM 2 
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EXPERIENCE. RESULTS. 
The Aegis Weapons System 
e Aegis is the world’s premier naval surface 
defense system. It is capable of 
simultaneous operation defending against 
air, surface, subsurface and ballistic missile i 


threats. 


e There have been 7 major releases, known 
as baselines, in the nearly 4 decade history 
of Aegis. 


e Recent baselines have focused on re- 
engineering the weapons system to take 
advantage of commercially available off- 
the-shelf (COTS) operating environments 
(OE). 
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EXPERIENCE. RESULTS. 


A Migration to Open Architecture 


Open Technology 


rg 


= fT 


Manufactured hardware and 


developed software Emphasis on COTS hardware 


, and software integration 
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Open Technology 


Sun. [eee S concurrent 
intel. & &® redhat Q_ 


invent 


e An Aegis ship not much different from a large-scale 
commercial data center. 


e The weapons system is comprised of a standard operating 
environment with unique components not seen in 
commercial architectures. 
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Application Management Equipment Management 
* Manage where applications are e Node/Server Management 
running — Diagnostics 
. Manage runtime state of the — Performance Monitoring 
applications 


¢ Network Management 


¢ Manage recovery and : 
reconfiguration ) sialic 
= 1 ti aon ti 
¢ Assess health status of the een ane Serna 
applications — Software Distribution 
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Root Cause Analysis 
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Aegis System Management 


e In the current baselines, 
System Management for the 
Aegis Weapons System is 
comprised of multiple 
components. 


e Each component exists as a 
standalone entity. 


e Together, they interoperate to 
provide an end-to-end solution 
for Operational Readiness 
Assessment of the combat 

system. 
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Aegis Weapons System 
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What is Insight? 


e An integrated computing system management toolkit. 


e It is a highly configurable suite of open source, 
commercially available, and developed tools that perform 
system management functions across the enterprise. 


— Hardware and software diagnostics 
— Performance monitoring 

— Network management 

— System control 

— Validation and verification 


3/16/20! 


EXPERIENCE. RESULTS. 


07 1:26:30 PM 11 


ror 
Ca ad the 
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The Catalyst For Insight 


2000 

EE : Deployment of Baseline 6 is 2002 ; : 

Baseline 6 signals the plagued by schedule delays Initial version of Insight 
emergence of COTS a Hoe tools with wealtanit deployed to local data centers. 


pened ipe oo Be heat t the detection and inefficient fault 
ops P . isolation. 


1970 - 1990 1995 2000 2001 2002 2003 2005 


1969 e02 3 2003 - Current 

ghey Baseline 7 signals the ute Papel Insight is deployed to 

emergence of an all-COTS an Baseline 7 for use in shipyard 

COMpULEM suite. 7 oe integration activities and 
shipboard Operational 
Readiness Assessment. 


Aegis begins. Baselines 1 
through 5 are deployed with a 
proprietary operating 
environment. 
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Insight Component Architecture 


e Framework 


— Graphical User Interface, 
Network Services, and Remote 
Agents 


e Tools 


— Configurable collection of “best- 
of-breed” products and utilities 
to perform system management 
functions 


¢ Configuration Data 


— Easily tailors Insight to the 
needs of the open architecture 
enterprise being managed 

¢ Baseline Data 
— Repository of expected OE configuration state information 
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Extensive Use Of Open Source Technology 
e Over 40% of Insight is 7 
comprised of Open Source Functional Composition 


software 


—Permits selection of cost 
effective, best-of-breed 
solutions 


— Reduces development time 


— Leverages intellectual 
resources from the world 
wide development 
community 


or-Telirclipdlale me) ame) el-)amevelulge-mcr- Wile 


approximately $1M in development costs 
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Application Management Equipment Management 


e Manage where applications are e Node/Server Management 


running — Diagnostics 
* Manage runtime state of the — Performance Monitoring 
applications e Network Management 
e Manage recovery and : 
reconfiguration Asset Management 
— Validation and Verification 
¢ Assess health status of the aati oe 
applications — Software Distribution 


Fault Detection / Fault Isolation 


Root Cause Analysis 


Insight’s Toolkit 
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Insight Functionality 


Performance 
Monitoring 


¢ On-demand Testing 
¢ Runtime Monitoring 
¢ Interactive Control 


System 
Control 
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Consistent Interface for Disparate Tools 


e Standardizes test execution 
¢ Hides tool variability from the user 
e Provides common results 


ak=jfelalm-jcclalercle|p4-t-m alm ->.<cceeld(olamr-laleMe-t-10 || come) m-r- (ela cole) B 
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Hardware and Software Diagnostics 


e Fault Detection / Fault Isolation (FD/FI) 


e Assess the operational state of hardware and software 
components 


—NTDS Devices 

—1/O Boards 

— Specialized Processors 
— Tactical Interfaces 

— Middleware Components 
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Increased Productivity 


Integration Productivity Rate 


100% 


The availability of Insight has increased ae 


the productivity rate to over 90% by 
providing rapid identification of operating 
environment issues. 40% 


60% 


20% 


0% 
2001 2002 2003 


Rapid Resolution = Increased Productivity = Significant Savings 
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Performance Monitoring 


e Assess infrastructure performance under operational 
conditions 


—CPU utilization 
—Memory utilization 
— Process activities 
— Disk usage 

—1/O activity 
—Kernel-level trace 


e Data is represented in real-time as well as archived for 
Statistical representation 
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Insight assesses operational 
readiness of combat system A 
components from a single = 
console 


Powermax Se Linux 
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Validation and Verification 


e System Operability Test 


e Checks a number of Operating Environment configuration 
parameters to determine overall system operability 


— Configured Devices 

— Filesystem Integrity 

— Kernel Tunables 

— Network Tunables 

— Installed Packages 
—OS Version and Patches 
— Firmware 


e Ensures that the current state of a host’s OE Is consistent with 
established baseline data 
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Automated Validation of the Operating Environment 


Integration and Test Facility Deployed System 


Tatsdlelajmclarcvelactmaar-tmeal-me(-yel(o\\(-lemere)atilelele-iacela 
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Automated System Validation: Months to Minutes 


There are over 6000 
configuration 

parameters on each 
node that are optimized 
for the application 


Manual validation of 
the configuration for 
the entire computer 
suite would take 

nearly 4 staff months 


Insight provides 
accurate validation for 
the entire computer 

suite within 3 minutes 


PNUive)aat-icclemvc-licer-iece)am e)genvace(ct-mere)anle)(-itcmere)avice(-)arer-mlamanl-me(-)e)(eV(-10| 
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Network Management 


e Assess health status and performance of the local area 
network 


—LAN operability 
— Routes 
— Network utilization 


System Control 


e Manage the run-time state of the tactical applications 
— System Initialization 
— Shutdown 
— Reboot 
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- == Run-time state of the system = = 
| , _ can be monitored and 
| controlled from a single console | 


af a 
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Concept of Operations 


e Insight is used throughout the life-cycle of the combat 


system. 
Development Data Staging and Test : . 
Center Facility Shipyard Integration Deployed Systems 
i 


Runtime status 
System validation, diagnostics, operability tests monitoring, operability 
tests, diagnostics 


Validation of the build 
environment 
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Safety Checks 


Insight’s safety agent prevents 
intrusive functions during tactical 
Operations. 


awe 


“ 
kK 
a 
= 
n 
Ww 
[a4 
wi 
UO 
Zz 
Ww 
[a4 
Ww 
oO 
x 
ff 


The Benefits of Insight 


EXPERIENCE. RESULTS. 


A Vital Maintenance Solution for 
the Aegis Weapons System 


| -, 
| 


Proprietary systems had an 
available toolset. 


Insight provides a 
robust toolset for COTS 
systems. 
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Applying Insight to Support Various Activities 


Development 
Engineering activities 


Life Cycle Support 
Fleet maintenance operations 


Shipyards 
Integration activities 


Training Center 
Aegis curriculum for sailors 


Sailors 
Shipboard system management 


Insight is a key component for Aegis System Management 
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Enhanced Operational Readiness 


Dispatching engineers to Delaying a missile test due to Meeting deployment schedules 
troubleshoot shipboard system instability: and increasing credibility with 
issues: your customer: 

$75K $1M Priceless 
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“Innovation: Delivered” 


¢ Deployed for 3 % years 

¢ Multiple Baselines 

¢ Domestic and Foreign Test Sites 
¢ Domestic and Foreign War Ships 
¢ Certified for Tactical Operations 
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THE OPPORTUNITY TO MAKE A DIFFERENCE HAS NEVER BEEN GREATER 


Tradespace Analysis for Capabilities, Effectiveness, 
and Resources 
GR RVAN@i=159 


Agenda 
¢ The Case for Integrated Analysis 
° What is TRACER? 


« Example TRACER Analyses 


¢ The Way Ahead 


Why Early Tradeoff Analysis? 


Ability to influence 
the process 


Life-cycle cost 
determination 


Cumulative Percent 


Ss Production 


Cc 


Traditional Alternatives Analysis Approach 


Evaluates limited number of alternatives 


e Stovepipes reduce incentives to share insights across domains 


Takes lots of time and resources to evaluate each alternative 


Does not facilitate finding optimal solutions 


fr 1% 
ee ee 


RDA Operational Resourcing 
Domain Domain Domain 


MSA MS 


Alternatives 


Time 


Integrated Alternatives Analysis Approach 


Exposes the entire tradespace 


Integration enables insights to be gained through the synergy 
of the domains 


Greatly reduces the time and resources for comprehensive evaluation 


Embedded analytical tools facilitate finding optimum solutions 


Tradespace 
Alternatives Integrated Outcome 
Analyses Visualization 


What is TRACER? 


° TRACER facilitates the application of systems engineering to 
acquisition analysis 

* TRACER implements a holistic investment analysis approach in an 
analytical framework that: 


— simultaneously addresses multiple evaluation disciplines including 
warfighting effectiveness, performance, and cost 

— can be applied at different levels in the resource analysis environment: 
strategic, operational/tactical, and acquisition 


— aims to explore the widest possible solution space, informing the 
alternatives analysis with cost tradeoffs 


* TRACER enables analysts and decision makers to do Cost As an 
Independent Variable (CAIV) analyses 


¢ TRACER methodology uses modeling and simulation to reduce the 
time, resources, and risk associated with investment decisions 


Example: JTRS Tradeoff Analysis 
How much JTRS capability 


Increase or reduce ia 
# of channels 


TRACER 
Integrated 
Analysis 


Data Rate 
(Kbps) 


Raise or lower 
the frequency (MHz) 


Increase or decrease 
power out (Watts) 


Multi-dimensional Tradespace 


To maximize decision tradespace, optimize both within and across dimensions 


Solution 
Evaluation 


Strategic 
Investment 


Alternative 
Designs 


Alternative 
Concepts 


Capabilities 


Breadth of tradespace 


Cost & 
| Effectiveness 


Performance/ 
Life-cycle costs 


Resource 
Requirements 


ET > 
High Decision-making levels Low 


Multi-dimensional Optimization Problem 


To maximize decision tradespace, optimize both within and across dimensions 


Strategic 
Investment 


Concept 
Evaluation 


Alternative 
Designs 


Alternative 
Concepts 


Capabilities 


Breadth of tradespace 


Cost & 
| Effectiveness 


Performance/ 
Life-cycle costs 


Resource 
Requirements 


ET > 
High Decision-making levels Low 


Operational Example: 


Air-to-Air Campaign 
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Air-to-air Campaign Analysis 


Requirement: Blue fighter fleet must defeat an optimal assault 
from a mix of 792 Red aircraft, allowing no more than 1,100 
tons of bombs dropped by “leakers,” with 80% confidence 
that there are no more than 45 Blue losses. 


Current fighter fails: 1,200 tons dropped, 56 Blue losses 


Approach 1: Improve airframe and propulsion (at cost of 
$12B) 
— Result? Success! 1,067 tons dropped, 42 losses 


But, how else might the requirement be met? 


Approach 2: Improve radar (cost and outcome are functions 
of improvement) 


Approach 3: Improve missile (cost and outcome are functions 
of improvement) 


PAGE 11 


Air-to-air Campaign Analysis Framework 
Outputs 


><"SLAACM DOE Demo 


LMI Radar LMI Kill Rates LMI SLAACM 
Model Model 


SLAACM = Stochastic Air Campaign Model 


PANGle il 


Air-to-air Analysis Design of Experiments (DOE) 


Choose 492 randomly distributed values for Power-Aperture 
ratio and SSPK ratio 


P-A ratio values range between 1.0 and 5.0 


SSPK ratio (improvement over current performance level) 
values range between 1.0 and 1.537 


TRACER Analysis 


Run all 492 cases and graph results 


Measure outcomes in blue losses and bombs dropped, and total 
modification/acquisition costs 


Identify best solution among the feasible outcomes 
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Air-to-air DOE Results 


Modification Cost Bombs Leaked (tons) Blue losses 


Power-Aperture Ratio Power-Aperture Ratio Power-Aperture Ratio 
2 2, ae ee 


71.5 
r142 se 
H13@ 
“12% E 
4.10 = 
1.0 


rT. ca. © . oo. 7. 
12 3 4 § 
Power-Aperture Ratlo 


3 4 
Power-Aperture Ratio 
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What's Next? 


- Tailor framework to assess systems engineering disciplines 
— Develop parametric cost and effectiveness relationships for each 
discipline 
— Use TRACER analysis approach to provide insights into the costs 
and effectiveness of combinations 


Design 
Engineering 


| Reliability 


Safety 


Human Systems |, 
Engineering 


Interface 
design 


Software 
Engineering 


Trades 


ve 


System of Systems 
Engineering 


| Miscellaneous 
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Contacts 


Dr. Pete Kostiuk 
Program Director for Resource Analysis 


pkostiuk@I|mi.org 
(703) 917-7427 


Gerry Belcher 
Program Manager for Cost Analysis 


gbelcher@Imi.org 
(703) 917-7073 


Kevin Anderson 

Technical Lead for TRACER 
kanderson@I|mi.or 

(703) 917-7033 
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Test and Evaluation of 
Electromagnetic 
Railguns 


NDIA T&E Conference 
March 12-15, 2007 


DISTRIBUTION STATEMENT A. Approved for public release; distribution is unlimited 
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INR EM Railgun - Game Changing NAYSEA 


Revolutionary Research .. . Relevant Results 


ABOVE SENSIBLE ATMOSPHERE 
Jian] )ilicssomelsrere) avilleiiiey a 


| Hypervelocity Impact 
% (MACH 5.0) 


(MACH 7.5) 


Indirect Fire (200+ nm in 6 minutes) 


Fixed and Relocatable ; 
Targets at Long Range 


_ @ Long-Range e Large Capacity Magazines 


| @ Time-Critical @ No Propellants = 
) Ape (24/7) ® ine re eee es, Sapa it Fs a Fs, * Se ae 
_ @ All-Weather e Increase ip Design Options rt eae ie . = 
' e No Unexploded e Reduced Ship Vulnerability ===)". Support for Distributed Ops ==" == 
Ordnance Issues e Leverages Navy Investment in = SS SS eS SS SS Se 
Integrated Power System SSS —— SSS SSS Se 
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Fe 
4 
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PS 


Lorentz Force = Current (J) X Magnetic Field (B) 


or 


Lorentz Force =1/2 Inductance Gradient (L’) * Current (I)42 
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Current Facility NAVSEA 
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Key Statistics (2007) Temporary 


¢ Operating Muzzle Energy: up to 7 MJ oe Vans 
¢ Terminal Area: 16-MJ Slug 


¢ Gun lines: 1 
32-MJ Green 


Farm PFN aon 


<+ 


s+ 


8-MJ SSG ——____g 
BA: , 
4 un Line #1 


Terminal Catch: Up to 
16-MJ Slugs 
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Spark Gap Switch 


Slide 7 


Sr. SSG Construction NAVSEA 
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Safety Shield 
(capable of stopping broken studs) a 


Containment Studs me 


Bore 
Insulators 


Upper 
Containment 
Plate (3 total) 


\\ 


Side 
Containment 
Plate (8 total) 


; ; Lower 
a: jr ; Containment 
\ od Plate (3 total) 
\ ; LS 


Rail Backing 
Insulators 


Pre-Injector 
Tube 


Launch Package 


Rails 
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Total Mass = 2.3-3.4 kg 
Aluminum Slug and 
Armature 

Nylon Bore Riders 
Design based on earlier 


work at Kirkcudbright and 
Greenfarm 
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Muzzle Chamber 

¢ 1" Thick A36 Steel Plate 

* Bolts Directly to Gun Foundation 
* Bolts Directly to Bridge Section 

¢ Adaptable to Variety of Launchers 


c 


c 


Bridge 
Section 


*«eecceccecceecctcecte 


Muzzle Chamber 
Assembly 


Recoil Plates 
°3" Thick A36 Steel Plate 


*Bolt Directly to the Gun Foundation Plates 
*Bolt Directly to Underside of SSG 


Slide 10 


Terminal Area Design NAVSEA 
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20 Each NRL Shield Blocks - 


7 Each Sand-Filled Steel Boxes Double Course 
(4x 4’ x 3’) . (8’ L x 2.5’ W x 1.5’ Thick) 
(21' of Sand Along Line of Fire) Interlocking, Reinforced 


7 Each Bridge Sections 
(8’ L x 16’ W x 10’ H x 10” Thick) 
Free Standing, 


Reinforced Concrete 
; ~75” Above the Ground 
Flash X-R 
Boney CUlOUrS ~66’ to 15t Sand Box 


NS ~37 ft Opening 
an _—= 


Ne is 
a 


ee Concrete Blocks 
Statically Support last 3 


Sand-Filled Steel Boxes 


Concrete 


Shot Line 


24 ft 


l-Beam Catch Cart 
(8 mini-RR wheels, 12’ long) 
Move/Replace first 4 Sand- 
Filled Steel Boxes 


Terminal Pad 
(100° L x 32’ W) 
Standard Gage Crane Rails, 
3 Rows of Tie Downs every 8’, 
2 Instrumentation Troughs 
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Steel Muzzle 
Chamber 


Catch Component NAVSEA 


: oe : : = muh WARFARE CENTERS 
evolutionary Research ... Relevant Results DAHLGREN 


e 7 Each Sand-Filled Steel Boxes, 
Total of 14 On Hand 


- 4fx4ftx3ft 
—- Wt 5740 Ibs when Filled 
- 21 ft of Sand along Line of 
Fire 
- Open Top, Stackable, 
4-Way Forklift Entry 
e |-Beam Catch Cart 
— Support the First 4 Sand 
Boxes to Allow Quick 
Movement & Replacement 
— Runs on Crane Rails Using 
Mini-Railroad Wheels 
e Concrete Blocks 


—- Support the Last 3 Sand 
Boxes 


I 

ie 
- 

i 
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Velocity — km/s 


Power Ramp Up Testing Plan (2.4kg) 


Banks 


Energy — MJ 


Velocity — km/s 


Actual Power Ramp Up Testinc 


Test Results 
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Video of Test Results 
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Charge Peak Muzzle Muzzle 
Mass Voltage Current Velocity Energy’ Efficiency 
Shot (KG) (KV) (MA) (m/s) (MJ) (%) 
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Charge Peak Muzzle Muzzle 
Mass Voltage Current Velocity Energy’ Efficiency 
Shot (KG) (KV) (MA) (m/s) (MJ) (%) 
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Shot 13 Breech Current and 
Muzzle Voltage war eae 


NAVSEA 


Actual Current 
Predicted Current 
Muzzle Voltage 
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Shot 13 Velocity NAVSEA 
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Muzzle Launch View 
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e Shot 7 
e Muzzle Arc is 500K Amps at 2.3 KV 
e 9 PSI Overpressure at 99” from muzzle 
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Flash X-ray | mages 


DAHLGREN 


Top View 


Side View 


Static Xray Image 


Shot 2 Xray Image Shot 4 Xray Image 


All images are 3 feet from muzzle 
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In-Flight | mages 
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Shot 8: 


Shot 9: = 


Shot 10: 


Shot 21: 


Tue Jan 30 2007 16:00:09,355 793 
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S Bore Life 
wi! EMLF Testing Concept wi 


DAHLGREN 


FYO? ees - (6: gee ees = (6: eels 3 4) mee ve 


Go No-Go Older Coe Potential 32 MJ 
ae a A 16-32MJ Testing Y A A100 Shot INP Demo__OPtion_@™ 
18 months 
Series A (SSG 2Q/3Q Series U (LL32 months 2Q13Q on 
Core ¢ Half mass-full velocity 


(10kg, 2.5km/sec) 
¢ Full mass-full current-2/3 velocity 
(20kg, 5.5MA, 1.7 km/sec) 


Te INP DEMO - 32MJ 
20 Shots es _ Analyze¢ 
MILCON : 100 Shot 


SSG - Army Single Shot Gun 
LL32 - 32MJ Lab Launcher 


40 Shots 
GA DEMO - 32MJ 


GA Adv Containment 


Adv Containment Launcher 


Contractor Tests 


on Gun-line #2 BAE Stub Tube BAE DEMO - 32MJ 


Adv. Containment Launcher A A A 
BAE & GA PDR CDR FAB 


Adv Containment Launcher 
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S&T Technology Challenges 
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e Launcher 


Multi-shot barrel life 


Barrel construction to contain rail repulsive forces A, 


Scaling from 8M] (state of the art) to 32M) > 64M) Muzzle 
Energy 
Thermal management techniques 


e Projectile 


Gun launch survivability (45 kGee acceleration, > 
Electromagnetic | nterference Potential) | 


Hypersonic guided flight for accuracy 
Lethality mechanics 


e Pulsed Power System 


Energy Density 

Rep rate operation & thermal management 
Switching 

Torque management and multi-machine 
synchronization (rotating machine) 
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Program Initial 8MJ Test Initial 32MJ Test Go - No Go 
Initiation Capability Capability Decision Point 
Aug 05 Aug 09 


BAE gry Bore “ © 32 MJ LAUNCHER 
ire Vevelopmen 
32MJ Lab Gun For P 100 SHOT BORE 
Bore Life Development LIFE DEMO 


General Atomics 
Advanced 


Containment 


Technology Development & Detail Design & 
Design Pralininan Desun Fabrication ©: MJ ADVANCED 

CONTAINMENT DEMO 

Launcher Testing 


& 
> > = 
P< ag: wre 
j ? ‘ j- 


EMLF Test Facility <i jas agit it i 


NSWCDD iy 33MJ 3°" 33MJ | )8P" 33MJ Advanced Capacitor Development 


Exercise fi 


100MJ Capacitor Bank For 


Executive Steering Committee Conclusion and Recommendation 


Integrated Launch Package (ILP) Development 


- General Atomics a 

5 Ya Rotating Machine Component Development 
Ps 3 — Boeing 

=z ——— ~_ Draper 

=} 

= 


Projectile Trades & Develop 4 Year Plan 


Milcon Addition 


WARFARE CENTERS 
DAHLGREN 


tere Protected 
OVS elis tek ANIL) P306 FY09 MILCON ($9.9 M) Control Room 


¢ Muzzle Operating Energy: 32 MJ x = 


¢ Terminal Area: 64-MJ Projectile 
¢ Gun lines: 2 


a — 100+ m3 Pulsed 


Power System 


Gun Line. #2 Gun Line #1 


Ballistic Tunnel 


Terminal Catch: Up to 
16-MJ Slugs 


Terminal Area: 
64-MJ Projectiles 
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QNR  Railgun Contact Information Miz 
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ONR 
Dr. Elizabeth D’Andrea (Program Manager) 


Office of Naval Research (Code 352) 
875 N. Randolph Street 

Arlington, VA 22203 

703.588.2962 


NSWC 


Mr. Charles Garnett (Program Manager) 
Naval Surface Warfare Center, Dahlgren (Code 308) 

6096 Tisdale Road 

Dahlgren VA 22448-5156 

540.653.3186 


Mr. Tom Boucher, P.E. (EMLF Test Director) 
Naval Surface Warfare Center, Dahlgren (Code 606) 

18236 Thompson Road 

Dahlgren VA 22448-5116 

540.653.6273 
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Revolutionary Research... Relevant Results 


Back-up 


ONR Navy Electromagnetic Railgun 224 


WARFARE CENTERS 


Revolutionary Research... Relevant Results DAHLGREN 


Why is it important? 

eVolume & Precision Fires 

¢Time Critical Strike 

eAll weather availability 

eVariety of payload packages 

¢Scalable effects 

eDeep Magazines 

¢Non explosive round/No gun propellant 
- Greatly simplified logistics 
—- NoIM (Insensitive Munitions) 


Issues 
eMissile ranges at bullet prices 
i Who needs it? 

What - It? , od eMarines and Army troops on ground 
Gun fired with electricity rather than *Special forces clandestine ops 

gunpowder “GWOT 
eRevolutionary 250 mile range in 6 Suppress air defenses 

minutes 


eMach 7 launch / Mach 5 hit 

eHighly accurate, lethal GPS guided 
projectile 

eMinimum collateral damage 


When? 

Feasibility Demo 2011 
«System Demo 2015 
«lOC 2020-2025 
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Launcher Projectile 


Pulse Forming 
Network (PFN) 


Capacitors or Rotating Machines 
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Key Parameters for Sizing a 


Revolutionary Research ... Relevant Results N aVa | = M [ eS U nc h eC r 


Pulse Forming 
Network Size 
1/* Launch Mass * Muzzle Velocity? 
Desired Muzzle Energy 


Current Bore Size & 
Profile Shape 
- Rail Separation — Launcher 
Forces Efficiency 
ec iledirete Barrel Length 


Heating - Max Projectile Acceleration 
- Bulk Rail Heating 
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Risk Ranking & Key Impacts 


Electronics 
Hardening 
Tactical ES 


ainment 
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Energy pba 
Density ‘ 
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ER4 otor ER2 
Switch Launch 
egg 4 y > lll 
>-E> 
Muzzle Rep-Rate 
: aie od 
Energy o 9% Dynamic Power 
“S &Ring | Management 
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vs Generation 
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Consequence of Sie C, 
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A Launcher Failure 
Impacts 


[| Projectile Capability 


Rotating Machine PPS ; 
Failure 


Impacts 


Probability of Failure, P, 
fe) 


Weight & 


© 
€®) Capacitor PPs Volume, 
Cost 


Ship Integration 


Revolutionary Research ... Relevant Results 


QIN ONR INP Phase 1 Objectives [21 
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e Traceability to 64M), 6-10 round / min indirect fire weapon 


system 

Bore Life 

— 32 Mega-J oule (Muzzle Energy) EM Lab Launcher 

- 10kg launch package; full muzzle velocity of 2.5kn7/sec 
—- 20kg launch package with full current of ~5.5MA 

— Demonstrate more than 100 shot bore life 

Containment 

— 32 Mega-J oule Advanced Containment Launcher 

—- 10kg launch package; full muzzle velocity of 2.5kn7/sec 
20kg launch package with full current of ~5.5MA 

—- 1000+ round predicted containment structural barrel life 
— Design for thermal management at a rate of 6 round / min 
— Design launcher for minimal round dispersion 

— Transportable on pallets and/or in sea containers, 

— Consider marine environment 
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= Integrated System Demo Strategy hZ\7z.1 


INP | 2005-2011 INP I] 2011 - 2015 


EM Lab Gun for Bore 
Life Development 


“~~, INP Phase | EM Railgun 
Demonstration Launcher 


INP Phase II Long Range 


wer INP Phase II EM Railgun Integrated System Demo 
Ne Demonstration Launcher 
Wa Launcher 


Technology — 
Launcher acre Dev. — 2—_-...—_—> 
Trades & => 
Concept Dev. es 
7% 
/ \ 
/ \ 
_ i \ 
| 


> = 
= —* : / Endo-Exo 
Projectil ILP Interface . H Testing 
rojectile in bore dynamics Mid;Rang a x" 


Trades & 
Concept Dev. 


~—-Terminal 


Fa Maclale[o 


Electromagnetic 
Test Facility 


Terminal High Velocity 
Powder Gun 
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NAVY ARMY 


lahs stan TESTING ($$$) 


SUB-SCALE TESTING ($$) 


BASIC RESEARCH 


° Auuisony Group 


¢ Spans Basic Research to Full-Scale Demo’s e Supports both Navy and Army 


e Parallel development paths via multiple EM Efforts 

research sites ¢ Government purpose data 
e Avoids Duplication rights to permit competition 
¢ Efficient use of test resources during the acquisition phase. 


Coordinated Development! 
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Revolutionary Research ... Relevant Results 


Lab Launcher - EMTF PNONElarecremOceniclinliilcilm mclelaealae 


| 


a. Phase of Project Period 
PPS iia 


Option Detailed Design, 
ll Fabrication 
and Demonstration 
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Advanced Containment Launcher iia 
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evolutionary Research ... Relevant Resu DAHLGREN 


Phase Phase of Project Period 


Basic Conceptual Design Trade 7 mos. 
Studies 

Army Trade Studies for Army 3 mos. 

Add Application 

Option! | Technology Development 30 mos. 
and Preliminary Design 

eae Detailed Design, 


Fabrication 
and Demonstration 


General Atomics Team Northrop Grumman Team BAE Team 
Q SOEING ( B communications 
a 


§.. RTA 


INP 
SAC =O EMD 


Jackson Engineering 
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Description of Effort 
¢ Develop long range projectile concept 
- Lethal 
- Consistent with Navy CONOPS 
- Compatible with any EML gun development 
e Identify critical development 
- GN&C 
- Aerobody (drag and thermal protection) 
- launched survivability 
e Produce a development plan 


The Boeing AASP Team Draper Team 


0 ©) TEXTRON Systems 


ATK K. > | =e 
Communications™ 
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Revolutionary Research ... Relevant Results 


e Rotating Machine 
— Watch Army Effort (Demo in FYO8) 


- Navy Specific Critical Component 
Development 


e Advanced Capacitor 
— Increased Energy Density 


—- Thermal Management for Multi Shot 
Operation 
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e Steel Muzzle Chamber Holes to Allow Bolting to 
Threaded Inserts Cast Into ae 
Bridge Secti 7/8"-9 Thread * Precast 
- Mates to both SSG & | Bridge Section| read) once bridge 
Lab Launcher Grade 8 Section 


Bolts 
— Bolts to 1% Concrete 


Bridge Section 


¢ Collar Plates Seal Gaps 
between Launcher & 
Chamber 


to Adapt to 
Each 
Launcher 1" A36 Steel Plate 8" x 6“ x 1" 


Steel Angle 


a! 
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Using Mission Essential Task Lists as a 
Basis for Mission-Based Operational 


Test Planning 


CJCSM 3500,04D 
1 August 200! 


UNIVERSAL JOINT TASK LIST 


Air Force Task List 
AFTL) 


() 
ane 


JOINT STAFF 
WASHINGTON, D.C. 20318 


Mr J.D. Carpenter 
Senior Military Analyst 
AVW Technologies, Inc. 


Using Mission Essential Task Lists as a 
Basis for Mission-Based Operational Test 
Planning 


Outline 
Introduction/Background 


Missions, Operations, Tasks, and Task 
Lists 


Describing a Method 
Lessons Learned 


Planning Meaningful Operational Tests 


DOT&E Priorities 


OFFICE OF THE SECRETARY OF DEFENSE 
1700 ORFENSE PENTAGON 
WASHINGTON, DC 20301-1700 


October 23, 2006 


MEMORANDUM FOR DOT&E STAFF 


SUBJECT: DOT&E Pronities 


In my short time here, it has become clear to me that DOT&E does exceptionally fine 
work to support the warfighters. | bope 10 build upon that reputation and establish an 
environment where each of you can continue to excel in our mutual effort to improve the 
nation’s military capability, As we work together, | want you to understand my general 
philosophy toward operational and live fire testing and evaluation, as well as my priorities for 
DOT&E 


General Philosophy: 


I believe we maust provide the Secretary of Defense and Congress, as well as other 
decision makers, with timely and objective evaluations and assessments of our weapon systems. 
Equally important, we have to provide commanders in the field with information about systems 
before they must rely on those systems in combat. 1 believe those evaluations should stan carly 
and that they contribute most to the warfighters when they identify system capabilities, which 
when integrated into the Joint force, enhance that force's ability to accomplish its missions. On 
the other hand, we must also identify system characteristics that may limit accomplishment of 
those missions of adversely affect life-cycle suitabilsty. 


Priorities: Consistent with the above philosophy, I want to focus on the following 
priorities: 


* Ensure the professional development of DOTAE penonne). We must ensure that our people 
are well trained and prepared to meet the challenges presented by the evolving acquisition 
and testing environments. 


Provide timely performance information to the warfighters. Given the current strategic and 
operational environments, the acquisition process is changing and the test community must 
adapt to those changes, While DOT&E must continue to support full rate production 
decisions, we must also be able 10 provide decision makers with assessments that help them 
make informed flelding decisions when systems are fielded for operational use prior to the 
full rate production decision. We must also assist the OTAs in ensuring our Joint warfighters 
and commanders are aware of system capabilities and limitations when sysicms are fielded 
carly. I believe these objectives are consistent with the congress intent of the FYOT 
National Defense Authorization Act 


Improve suitability. My impression is that systems generally scem to be doing well in 
enhancing mission effectiveness, The suitability of systems, on the other hand, requires 
additional focus. Effectiveness and suitability are not conflicting concepts if both are 
considered and assessed carly in the design and developmental processes. DOTAE must 


G 


locus the test community’s effort to identify failure modes and impacts early in these 
rocesses, It is far more important for a system to be effective when it is needed than when it 
is available. 


Enhance operational realism in early tests, including DT, With changes in the acquisition 
rocess focusing on developing and fielding systems on a shorter timeline, I see a need to 
incorporate operational realism into developmental testing to gain operational insights as 
early as possible. I believe there is more that can be done to synchronize DT and OT to 
enhance the discovery process during DT, eliminate surprises in OT, and shorten the overall 
development cycle. 


acilitate adequate OT resources, including adequate resources for the OTAs, It is important 
that adequate resources are devoted to test and evaluation within the Department of Defense. 
here must be adequate resources dedicated to OT&E to ensure test adequacy to determine 
operational effectiveness and suitability. I will champion the effort to ensure that we and the 
OTAs have adequate resources, including budget and personnel, to perform our Title 10 
responsibilities. 


While these priorities do not necessarily encompass all that we do, I believe that their 
implementation will move us along the path of continuing excellence, I welcome your 
comments on how these priorities can be improved because it is important that they become 


“our” priorities. 
Dr. Charles E. ig 


Director 


¢Provide timely performance 
information to the warfighters 


elmprove suitability 


eEnhance operational realism in 
early test, including DT 


eFacilitate adequate OT resources, 
including adequate resources for the 
OTAS 


So then, how do you inject realism into 
operational test planning? 


Relationship of Missions, Operations 
and Tasks 


MISSIONS, OPERATIONS, AND TASKS 


Missions | 


Extracted from CJCSM 3500.04D 


Other Important Terms 


Mission Task 

Mission Essential Task (MET) 
Conditions 

Standards or MOEs 
Supporting & Enabling Tasks 
TYPES OF TASK LISTS 


Mission Essential Task Lists (METL) 


Joint Mission Essential Task List (JMETL) 


PROVIDE AIR AND SPACE PROVIDE PRECISION 
SUPERIORITY AFT 1 ENGAGEMENT AFT 2 


Provide 
Counterair 
Capabilities 


AFT 1.1 


PROVIDE GLOBAL ATTACK AFT 4 


Provide 
Strategic 
Attack 
Capabilities 


AFT 4.1 


Provide the 
Capability to 
Ready the Force 


AFT 6.1 


Air Force Task List 


PROVIDE INFORMATION 
SUPERIORITY AFT 3 


Provide Provide Lethal Provide Nonlethal Provide Provide 
Counterspace Precision Precision CSAR Information 
Capabilities Engagement Engagement Capabilities Operations 
Capabilities Capabilities Capabilities 


AFT 1.2 AFT 2.1 AFT 2.2 AFT 2.3 AFT 3.1 


PROVIDE RAPID GLOBAL MOBILITY AFT 5 


Provide Provide Provide Special Provide Provide Air 
Counterland Countersea Operations Forces Airlift 
Capabilities Capabilities Employment Capabilities 

Capabilities 


AFT 4.2 AFT 4.3 AFT 4.4 AFT 5.1 


Provide Provide Air 
Refueling Spacelift Expeditionary 

Capabilities Capabilities Force (AEF) 

Capabilities 


AFT 5.2 AFT 5.3 AFT 5.4 


PROVIDE AGILE COMBAT SUPPORT AFT 6 


Provide the Provide the Capability Provide the 
Capability to to Prepare the Capability to 
Protect the Force Operational 


Environment 


Provide the Provide the Provide the 
Capability to Capability to Capability to 
Position the Force Employ the Force Sustain the Force Recover the Force 


AFT 6.2 AFT 6.3 AFT 6.4 AFT 6.5 AFT 6.6 AFT 6.7 


PROVIDE COMMAND AND CONTROL AFT 7 


Monitor Global Assess Global Plan Execute 
Conditions and Conditions and Military Military 


Events Events Operations Operations 


AFT 7.1 AFT 7.2 AFT 7.3 AFT 7.4 


Extracted from AFDD 1-1 


AFTL Tasks in Relationship to UJTL 
“Tactical Level of War” Tasks 


TAl TA) TA3 TA4 TA5 TAG 
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Conduct | Intelligence | Firepower | Combat Service Support | Command & Force 

Maneuver Control 
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Extracted from AFDD 1-1 


ART 1.1 


Provide Support 
to Situational 
Understanding 


ART 3.1 


Decide Surface 
Targets to 
Attack 


ART 5.1 


Conduct 
Mobility 
Operations 


ART 6.1 


Provide 
Supplies 


ART 6.6 


Provide 


Countermobility 
Operations 


ART 6.2 


Provide 
Maintenance 


ART 1.2 


Support Strategic 
Responsiveness 


ART 3.2 


Detect and 
Locate Surface 
Targets 


ART 5.2 


Conduct 


ART 6.7 


Survivability 
Operations 


ART 6.3 


Provide 
Transportation 
Support 


ART 1.3 
Conduct ISR 


ART 1.4 


Provide 
Intelligence 
Support to 
Effects 


ART 3.3 


Employ Fires to Influence the 
Will and Destroy, Neutralize 
or Suppress Enemy Forces 


ART 5.3 
ART 7.1 


Establish 
Command Post 
Operations 


Conduct 


ART 6.4 ART 6.5 


Provide Force 
Health Protection in 


Provide 
Sustainment 
Support 


ART 6.10 


ART 6.8 ART 6.9 


Provide General 


ART 7.2 


Manage 
Tactical 
Information 


Global Environment 


ART 2.1 


Perform tactical 
Actions 
Associated With 
Force Projection 
and Deployment 


ART 4.1 


Prepare to Defend Against Air 
Attack and Aerial 
Surveillance 


ART 7.3 


Assess the 
and Operations 


ART 7.7 


Support the 
Commander’s 
Leadership 
Responsibilities for 
Morale, Welfare, 
and Discipline 


Tactical Situation 


ART 7.8 


Conduct 
Continuous 
Operations 


ART 2.2 


Conduct Tactical C 
Maneuver Tr 


ART 2.5 
Occupy an Area 


ART 4.2 


Process Tactical 
Aerial Platforms 


ART 7.4 


Plan tactical 
Operations Usin 


the MDMP/TLP 


ART 7.9 


Develop and 
Implement a 
Command 
Safety Program 


ART 2.3 


onduct Tactical 
‘oop Movements 


ART 2.6 


ART 2.4 


Conduct 
Direct Fires 


Employ SERE Techniques 


ART 4.3 


Destroy 
Aerial 
Platforms 


ART 7.5 
Prepare for 
Tactical 


8 Operations 


ART 4.4 


Deny the 
Enemy Use 
of Airspace 


ART 7.6 


Execute 
Tactical 
Operations 


ART 7.10 


Conduct 
Public 
Affairs 
Operations 


Human 
Resource 
Support 


Provide 
Legal 
Support 


Provide 
Religious 
Support 


ART 6.13 


Provide Finance & 
Resource 
Management Services 


ART 6.12 


Engineer ART 8.1 ART 8.2 ART 8.3 


Support 


ART 8.4 ART 8.5 


Conduct 
Tactical 
Mission 
Tasks 


Conduct 
Support 
Operations 


Conduct 
Stability 
Operations 


Conduct 
Defensive 
Operations 


Conduct 
Offensive 
Operations 


ART 6.11 ART 6.14 


Conduct Civil- 
Military 
Operations 


Conduct Internment 
and Resettlement 
Activities 


Provide 
Distribution 
Management 


Provide 
Contracting 
Support 


Generated from FM 7-15 


Army Tasks In Relation to UJTL 
“Tactical Level of War” Tasks 


CJCSM 3500.04C 
FM 7-15 BOS UJTL TACTICAL 
TASK AREA 


ART 1.0-Intelligence BOS Deploy/Conduct Maneuver (TA-1) 

ART 2.0-Maneuver 

ART 5.1-Mobility 

ART 5.2-Countermobility 
ART 2.0-Maneuver BOS Develop Intelligence 

ART 1.0-Intelligence 
ART 3.0-Fire Support BOS Employ Firepower 


(TA-2) 


ART 3.0-Fire Support (TA-3) 


ART 4.0-Air Defense BOS Perform CSS & Sustainment TA-A 
ART 6.0-Combat Service Suppolt =) 
ART 5.0-M/CM/S BOS Exercise C2 TA-5) 
ART 7.0-Command and eae 
Protect the Force 
ART 4.0-Air Defense (TA-6) 


ART 6.0-CSS BOS 


Extracted from FM 7-15 


NTA 1.2 NTA 1.3 


NTA 1.1 


Maintain 
Mobility 


Move Naval Tactical 
Forces 


Navigate and Close Forces 


NTA 1.5 


NTA 1.4 


Dominate the Operational 
Area 


Conduct Countermobility 


NTA 3.1 NTA 3.2 NTA 3.3 


NTA 4.1 NTA 4.2 


Attack Fuel 


Targets 


Process Arm 


Targets 


Conduct Special 
Weapons Attack 


NTA 4.7 NTA 4.8 NTA 4.9 


Perform Civil Conduct Train 


Military Civil Forces 
Engineering Affairs and 
Support in Area Personnel 


NTA 4.14 


Provide Support Services 


NTA 5.2 NTA 5.3 NTA 5.4 


NTA 5.1 


Determine and 
Plan Actions and 
Operations 


Direct, Lead and 
Coordinate 
Forces 


Acquire, Process 
and 
Communicate 
Information and 
Maintain Status 


Analyze and 
Assess Situation 


NTA 5.7 NTA 5.8 


NTA 5.6 


Establish a 
Task Force 
Headquarters 


Provide Public 


Conduct Affairs Services 


Acoustic 
Warfare 


NTA 5.5 


Conduct 
Information 


Warfare (IW) 


NTA 2.1 


Plan and Direct 
Intelligence Operations 


NTA 2.4 


Conduct Analysis and 
Produce Intelligence 


NTA 4.3 


Repair/Maintain Equipment 


Personnel 
NTA 4.10 aad NTA 4.11 NTA 
Personnel 4.12 NTA 4.13 


Resource Operational Provide Conduct 
Managem Legal Advice Health Recovery 
ent Services and Salvage 


NTA 2.2 


NTA 2.3 


Perform Collection 
Operations and Management 


Process and Exploit Collected 
Information and Intelligence 


NTA 2.5 NTA 2.6 


Disseminate and 
Integrate Intelligence 


Evaluate Intelligence 
Operations 


NTA 4.4 NTA 4.5 


Provide Provide Transport Services 


Perform Provide 


Support 


NTA 6.1 NTA 6.2 NTA 6.3 NTA 6.5 


Perform 
Consequences 
Management 


Enhance 
Survivability 


Rescue Provide Security 
and for Operational 
Recover Forces and Means 


NTA 6.6 


Provide for Operational Safety of 
Personnel and Equipment 


Generated from OPNAVINST 3500.38B 


Why Use UJTL/AMETLs for 
Operational Test Planning 


“UJTL linkage to capability development processes enhances 
the identification of joint requirements, capability shortfalls and 
deficiencies.” (Extracted from CJCSM 3500.04D) 

Service task lists capture service doctrine and therefore is an 
excellent representation of required mission capability. 
Service task lists provide a concise picture of the major 
activities of a force (provide increased detail on what the force 
must do to accomplish its mission). 


Service task lists provide all the collective tasks possible for 
tactical units. 


So What? 


e Using mission requirements to assess the capability new 
systems will provide is a logical “next step”. 


¢ The ability to define capabilities-based requirements early 
in the acquisition process and to make better use of 
combined/integrated testing facilitates program manager’s 
delivery of effective and suitable systems to the warfighter 
with greater potential for reduced schedule and total 
ownership cost 


¢ Defining the capabilities-based requirements early on ina 
program also facilitates early test resource determination 
(i.e. facilities, test equipment, test objectives, data 
requirements, etc.) and more accurate cost projection 


Mission-Based System Operational Test Planning Process 


CONOPS 
ROC/POE 


UJ/ATLS 


Standards 


Conditions 


Mission METs 
Analysis KPPs 


Develop COls 
TM/DR 
Test Events/Test 
Test Methods 


Resource Availability . 

Establish System 
Development 
Schedule Test Schedule 


MA/COIs 
Conditions Write Test 
ats Plans 


Resources 


Lessons Learned 


Using the Universal Joint Task List/Service Task Lists to support mission 
analysis has proven useful in deriving mission essential tasks from which to 
develop mission-based COls. 


This methodology can be applied to any developmental system in any 
service. 


This methodology can be used regardless of whether or not 
combined/integrated testing is used 


This methodology would also prove useful for the developmental test 
community. 


Use of a database to manage data generated during this process is key to 
SUCCESS. 


An experienced analyst is worth his weight in gold. 


DoD policy guidance enhancement is required to fully optimize this 
methodology for integrated testing. 


Using Mission Essential Task Lists as a 
Basis for Mission-Based Operational Test 


ee 


Questions? 


Mr. J.D. Carpenter 
AVW Technologies, Inc. 
carpenter@avwtech.com 

(757)361-5830 


Back-up Slides 


Organization of Conditions for 
Joint/Agency Tasks 


1.0 PHYSICAL 2.0 MILITARY 3.0 CIVIL 
ENVIRONMENT ENVIRONMENT ENVIRONMENT 


4.1 Land el lee 3.1 Political 
be re Z3 Command, Control, & 32 ae 
1.4 Space Communications 3.3 Economy 
2.4 Intelligence 
2.5 Deployment, Movement, & 
Maneuver 
2.6 Combat 


2.7 Protection 
2.8 Sustainment 
2.9 Threat 

2.10 Conflict 


Extracted from CJCSM 3500.04D 


What is JCIDS 


Ensures the joint force has the capabilities to perform across the range of 
military operations 


Is a primary interface to the DoD acquisition system 
Implements an integrated process to guide new capabilities development 
A key linkage on how the future joint force will fight 


Provides the analytic baselines to support studies to inform capability 
development 


Leverages expertise to identify improvements to existing capabilities and to 
develop new warfighting capabilities 


Describing the Process 


Conduct mission analysis and identify specified and implied tasks contained in the guidance 
(ICD, CDD, CONOPS, etc.) 
— IDENTIFY THE MET - This will require thorough analyst knowledge of the system or OTA determination 


— DESCRIBE CONDITIONS - Conditions are used in the METL development process to express variables of 
the environment that affect task performance. 


— ESTABLISH STANDARDS - The final step in developing the MET involves selecting or developing 
performance standards consistent with the commander’s intent and CONOPs for a mission. 


— We found it useful to develop an Access database (or any other database product of your liking) that 
contains tasks and metrics matched from the Joint and Service task lists. 


— A database can be used to output tables that link tasks to CDD attributes as well as other references. 


Develop COls (mission-based from METs) 


— The UJTL is used to help frame COls in terms of the Joint and Composite Warfighter. The UJTL also helps 
define relevant Joint operational measures and conditions for the system under test. 


Determine data required to assess effectiveness and suitability of the system under test 
Define test methods to obtain required data 

Determine required resources 

Establish a test schedule 


Write the test plan 


— A database will prove useful not only in maintaining the linkage between tasks, metrics, test methods, data 
requirements and resources but can also provide the framework for required test plans. 


Demonstration 


¢ Process applied to System XX 
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nel Growth Planning Model SSPLAN 
Overview 

¢ Principal Benefits and Features 

¢ Model Inputs and Outputs 

¢ Procedure used to Generate Model Outputs 
Bombardier Applications 

¢ Bombardier Overview 

¢ Problem Statement / Opportunity 

¢ Reliability Maturation Model 

° New Reliability Growth (NRG II) Model Objectives 

° Railway example of implementation process 

¢ Interactions between basic elements of reliability process 

¢ Summary of NRG II Benefits 
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Reliability Growth Planning Model based on system level reliability 
objective (mean test duration between failure, MTBF) and coordinated 
subsystem growth program inputs. 


UNCLASSIFIED 


SSPLAN Overview 


Applies to systems composed of a series of subsystems that 
independently generate failures. 


Can accommodate a mixture of growth and non-growth subsystems. 
Measure of test duration, t , is continuous (e.g., time, distance ). 


Assumes for each growth subsystem i, the number of failures that occur 
in test interval [0,t], N.(t), is governed by a nonhomogeneous Poisson 
process (NHPP) with a power law mean value function A;tFi assumed at 
system level in U.S. MIL-HDBK-189. 


Developed by AMSAA and documented in AMSAA Reliability Growth 
Guide (Technical Report No. TR-652, Sept. 2000) 


UNCLASSIFIED : 
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Ve AMSAA 
Excellence in Analysis 


Principal Benefits and Features 


Helps construct a set of subsystem planning curves with associated 
subsystem test durations and target MTBFs 


¢ that are consistent with system reliability objective and growth 
subsystem reliability allocations 


¢ whose achievability can be gauged by past experience 


Subsystem planning curves and test durations have property that if 
realized during developmental test program, with a specified probability, 
subsystem test data would provide a specified level of assurance that 
system reliability objective is met. 


Subsystem planning curves support decision process with respect to 
allocation of test resources 


° prior to test program 
¢ during testing with regard to reallocation to address subsystem 
reliability deficiencies 
v can provide objective basis for prioritizing subsystem corrective 
action efforts 


System and associated subsystem planning curves serve as benchmarks 
against which reliability improvement can be measured 


¢ highlights to program management and customers assessed reliability 
versus reliability goals at program milestones 


Y at system and subsystem levels p 
UNCLASSIFIED 
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Model Inputs & Outputs 


= System Level Inputs 
> System MTBF objective, Moy; sy. 
° Statistical confidence level y for lower confidence bound LCB, 
on achieved system MTBF 


* Specified probability p, that subsystem test data would yield a 
value of LCB, that meets or exceeds M,,, 5), if subsystems 
grow in accordance to planning curves for planned test 
durations 


vy is the specified level of assurance 


Vp, is the specified probability that assurance level would 
be realized under growth assumption 


= System Level Output 
¢ System MTBF target , Miarg,Sys 


Y Reciprocal of sum of subsystem target failure intensities 


¥vM >M for practical values of y and p, 


targ,Sys obj,Sys 
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Model Inputs & Outputs 


=" Growth Subsystem Inputs 


¢ Growth rate a, 
Y Negative of slope on log-log plot of expected cumulative failure intensity 
versus Cumulative test duration 
Initial test period t, ; 
Y Growth is planned to commence by the end of initial test period 
- Average MTBF expected over initial test period, M, ; 
* Allocation fraction, a; , of growth subsystem portion of target system failure 
intensity to growth subsystem | 
v If all subsystems have growth programs, target failure intensity for 
subsystem satisfies Ata.g; = a; Marg,sys 
¢ Maximum subsystem test duration 


=" Growth Subsystem Outputs 


¢ Test duration for subsystem i, T; 
* Target MTBF for subsystem i, Mig; 
- Expected number of subsystem failures in test, E(N,) = T; / {(1-))"Miarg, 
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t, = Time of Initial Period 
M, = Avg. Initial Pd. MTBF 
Myarg= Target MTBF 

a = growth rate 


@ = growth rate 


Cumulative Failure Rate 


Cumulative Test Time 


Duane Postulate: If changes to improve reliability are incorporated into the design of a 
system under development, then on a log-log plot, the graph of cumulative failure rate 
versus Cumulative test time tends to exhibit a linear relationship (Duane, 1964). 


UNCLASSIFIED , 
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Excellence in Analysis 


Outline of Procedure to Obtain Subsystem Test Durations 
(case where all subsystems are growth subsystems) 


N,(t) is a NHPP with rate of occurrence function p,(t) 


Ss 


- p,(t) = A,B;t®i- where B; = 1-a; and A, = t,;“'/ M,, for growth subsystem i 
teps 

* Use trial value M4 sy, to calculate Ajogi = aiMtarg,sys 

° Obtain trial value T; by inverting eq. Ajagi = Pi(T)) 


¢ For each growth subsystem i simulate NHPP from 0 to T, 


- Calculate pseudo demo. test no. of failures np; = n,/2 & time Tp; = T;/ (2Be¢¢/) 
v B.sti is max. likelihood estimate of B, from simulated growth test data. 


v Equate point estimate and LCB on M,,,,; from pseudo demo. data to 
estimates from growth data to obtain pseudo demo. test data. 


- Combine subsystem pseudo demo. data to obtain approximate LCB, on M,., 5,5 
Y Applied Lindstrom — Madden method adapted for continuous test duration. 
Y Could use other methods for combining pseudo demonstration test data. 


> Repeat last 3 steps prescribed no. of times to estimate Prob ( LCB, 2 M4; sys ) 


¢ If estimated probability is close to py stop - the current trial T; are chosen as the 
subsystem test durations ; otherwise adjust M,,,,s5,, and repeat above steps. 
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AEROSPACE & RAILWAY INDUSTRIES APPLICATION 


A Successfully Implemented Coordinated Subsystem 


Reliability Growth Planning Approach 
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Bombardier’s Implementation/Customization of the SSPLAN — 
New Reliability Growth (NRG II) 


Company Overview 


= Corporate office based in Montreal, Canada 


= Workforce of some 55,800 people worldwide as at 
January 31, 2006 


= Revenues of $14.7 billion for fiscal year ended 
January 31, 2006 


= More than 94% of revenues coming from foreign markets 
= Listed on the Toronto Stock Exchange (BBD) 


BOMBARDIER 


The focus of Bombardier Inc. is based on two pillars 


Transportation Aerospace 


45% of total Revenues 55% of total Revenues 


Annual Revenues in (2005/2006): Annual Revenues in (2005/2006): 
US$6.6bn US$8.1bn 


«  BOMBARDIER 


*Figures for the year ending January 31, 2006 


Problem Statement / Opportunity 


= Problem Statement 
¢ Warranty costs and customer dissatisfaction 


¢ Weakness in the process to predict, optimize and govern the product 
reliability in order to meet Entry into Service and in-service 
performance. 


= Opportunity to raise reliability performance 
¢ Insure reliability performance meet commitments as per schedule 
e Reduce Warranty cost 
¢ Model Life Cycle Cost 
¢ Improve Maintenance Program 
¢ Highlight to senior management reliability progress of all subsystems 
¢ Prioritization of corrective actions 
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Reliability Maturation Model 


Calendar Date 
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= Potential shortfalls in managing reliability Oo 
¢ System Reliability measurements start too Late. ae 
¢ No Prediction of reliability at Entry Into Service(EIS). “a 
¢ No Proactive action to ensure that EIS reliability will be met. 


=" Need reliability growth program plan to conduct trade-off analysis 
between EIS calendar date and EIS expected reliability. 


= Need to measure against program plan. 7 BOMBARDIER 


New Reliability Growth (NRG II) Model Objectives 


= Predict Reliability Maturation 
¢ Based on past performance by system and supplier 
¢ Include reliability growth from the start of testing 
¢ Supports spares contingency requirements 
= Optimize lifecycle costs linked to reliability 
¢ Perform trade studies between increasing testing and fixing issues in the field 
¢ Allow analysis between EIS calendar date and EIS expected reliability 
¢ Optimize maintenance program 
= Govern Reliability Growth and issues from day one 
¢ Setup a framework to compare actual reliability to planned reliability 


¢ Comparison provides basis for efficient proactive management with regard to 
failure mode mitigation 


= Utilize coordinated subsystem reliability growth strategy 
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Classic growth — Planning and monitoring at system 
level only 


MTBF/MDBF 


Aircraft/Train 


1 Flight ae Reach Flight Hours/Kilometers 
or Objective «= ##BOMBARDIER 


1st Preseries 


NRG Il Approach 


Reliability Growth’ 
Monitoring 


on 
“I 


se 


g; ircraft/Train 


Reach Flight Hours/Kilometers 
Objective ‘©  BOMBARDIER 


st ; 
Component oie EIS 
Tests Or 


1st preseries 


Reliability Growth : method NRG Il 


¢ Name of the Product 
¢ MDBF or MTBF Objective (Contractual) 
¢ Confidence Level 

¢ Acceptance Probability 

¢ List of all Main Subsystems 


em |e || x 
Project ? 

Project properties 

Name: [European Train European Train 


Objective» 203! MDBF (Km) 3816 
Confidence level O80 
Acceptance Probabilly [98 SS 


Description |For Demonstration " 


B=] Serie 10 
B=] Serie 11 
N=] Serie 12 
F=] Serie 13 
B=] Serie 14 


Je 


x 


Project failure allocation % : 100 


ot Options 


Reliability Growth : method NRG Il 


¢ Growth rate alpha 

¢ Failure intensity allocation 
¢ Initial MDBF 

¢ Initial Test Distance 

¢ Maximum Test 


Project ? 
Sub System requirements 
Name: |5 8 


WEBS [optional] : | 


Reliability Growth Specifications 


( Growth c~ 


Growth Rate (0,22 
Failure Intentisity Allacation| 0,1434 


Simulations Specifications 


Initial conditions [need both] 
Initial MDBF (Km) 5938 Maximum Test (3000000000 


Initial Distance (Km) 1 7914 [oo —t~<“Ct~‘ts™S 
js 


Optional data 
Comments and/or Assumptions 


ot Options 


Reliability Growth : method NRG Il Per Subsystem: 


¢ Number of Failures 
¢Total test Distance 

¢ Final MDBF 

For the Product: 

¢ Computed Target MDBF 


level fems 


‘eB acc: ice ems | 
ae | ae) 14 


2s | c ; ‘imi 


ne 


a | - : } 4 re 
0.0027| 17.64] 17,055,812| 1,585.03 
| Growth _| 


4 
Growth 0.075 56,68 
0.044 95,74 
0.139 30,72 
0.080 53,29 

33,67 0.02 

33,45 0.0 79,25 
593 0.14 29,84 
0.22 : 19,03 
0.0597] 25.96 71,68 
S11 2.544000] 0.0015]  15.27|26,139,970| 2,853.07 
3000000000] -33,415| 0.114 37,47 
0.031 135,43 
0.003[ 21.95) 1,426,53 
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Example of Reliability Growth monitoring of initial subsystem 
MDBF’ s fora Product composed of 14 subsystems 


NRG II Method 


Using event 

~ #« recording per 
period versus 
Target on 
benchmark curve 


* 


Establishing priority of corrective actions 


Life Cycle Cost Reliability performance 


% measured LCC 
exceeds target 
LCC 


Reliability performance 


below 0 
>= target | to -15% | below -15% 
za wa 


Priority 
weight 


3 
Safety failures have the highest priority »  BOMBARDIER 


Interactions between basic elements of reliability process 


car 
aN 
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Summary of NRG II Benefits 


= Promotes a proactive approach to maturing product reliability 


= Establishes corrective action priority based on comparisons 
of measured LCC and reliability values to target values 


= Highlights to senior management reliability progress of all 
subsystems 


= Reduces cost of Product Introduction 
" Assists in modeling LCC 


= Assists in performing trade-off analysis between EIS calendar 
date and expected EIS reliability 


= Helps in optimizing maintenance program 


= Applies to new product development or in-service 
improvements 
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Chris DiPetto 
Deputy Director, 
Developmental Test & Evaluation 


OUSD(AT&L)/Systems & Software Engineering 


March 13, 2007 
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¢ Intro to OSD DT&E 


¢ DT&E Priorities 

¢ DT&E Technology Maturity 
Initiative 

¢ Plan of Action 


Secretary 
of 
Defense 


User Require- 
ments & Concept 
of Operations 


Requirements Integration 
& Architecture / & Test 


Component 
Integration 
& Test 


Component 
Design 


Procure, Fabricate, 


& Assemble 
Parts 
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Primary 

¢ DI&E Policy & Guidance 

¢ T&E Workforce Education 
Secondary 

¢ Acquisition M&S 

¢ Systems Energy Policy 

¢ DoD Acquisition System Safety 
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My Priorities... 
¢ Support Faster Fielding of Improved Capabilities 


¢ Reduce Risk of Immature Technology in 
Systems Development 


¢ Revitalize T&E Workforce Education 


¢ Promote Joint T&E in Live-Virtual-Constructive 
Environments 


¢ Provide Effective Acquisition Policy and Practices 
for DI&E 


& 
> 
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- 2006 QDR: “...a more effective acquisition system and 
associated set of processes.” 


¢ Acquisition goal - cut time in half, from 10+ to 5- years 


(1) reduce technical & programmatic risk, prior to program 
initiation 

(2) change people’s mind-set; focus on trading greater 
capability for earlier fielding 


New approach: evaluate cost, requirements, and technology 
alternatives - improved Concept Decision for MS A (“Big A”) 


Result: start programs with firm requirements & mature 
technology MS A 


Strategic . ‘ T MSB System MS C ; 
Planning ¢ aan eae & Development & Production & 
Guidance “ONCEPtS y RR Demonstration Deployment 


Assure testable requirements in Big “A” Eval of Alt’s / CD 


Include in request for proposal (RFP) T&E implications for 
System Development and Demonstration 


Fully integrate T&E strategy - CT, DT, OT 
Efficient test data philosophy: collect once, use often 


Not pass-fail; learn, define, understand system’s 
capabilities and limitations...for fielding at predefined time 


Operational environment and operators in DT, soonest 
Mutually supporting plans: 
— Systems Engineering Plan 


— Test and Evaluation Master Plan 


— System development Statement-of-Work and RFP 


Technical Maturity: System-level 


Integration 


Manufacturing 
Technology 


Maturity 


Reliability 


Resources 


¢ Studies find that immature technology is a primary source 
of cost and schedule risk 
— GAO -- DAPA 
— QDR -- SSE/AS Program Support Reviews 


¢ “Programs that started development with immature 
technologies experienced an average acquisition unit cost 
increase of nearly 21 percent” (GAO-05-301, March 2005) 


¢ FYO6, PL 109-163, Section 801 requires USD(AT&L) 
certification, before Milestone B, that “the technology in the 
program has been demonstrated in a relevant environment” 
— Above wording equates to Technology Readiness Level (TRL) 6 


PM chose “a software architecture that depends upon 
COTS middleware that does not yet exist “ 


— Although an alternative has been identified, no effort has been 
expended to pursue this solution 


“Technology maturity growth of critical Engineering 

Development Models lagging the plan” 

— PSR Recommendation: Initiate development of off-ramps to maximize 
operational performance 

“Technology Readiness Level (TRL) 6 of major 

subsystem at Milestone B is unlikely to be achieved; 

planned testing will not support accurate assessment of 

true maturity” 


“TRA conducted too late to influence decision process” 


Purpose 
¢ Add Technology Maturity focus into the Systems Engineering and DT&E 
processes to: 
— Reduce technical, cost, and schedule risk 


— Increase the rigor of SE 
— Plan for alternatives in the event of TM difficulty 
Verify TRLs during DT&E 
— Updates will complement proposed Risk-Based Source Selection, 
Time-Defined Acquisition, and Concept Decision (CD) processes 
Scope 
¢ Leverage existing acquisition review structure 


¢ Use existing DDR&E Technology Readiness Assessment (TRA) 
methodology 


Technical Review Decision TRL (min) 


Initial Technical Review CD ii 


Alternative System Review MSA 4* 


Systems Verification Review/ MS C 7* 
Production Readiness Review 


* Guidance, not statute 


Technical Review Decision TRL (min) 


Opportunities 
Evaluation of Alternatives EOA pp dra 
Alternative System Review (ASR) co 4-5*} Merged 


Systems Verification Review/ MSC 7* 
Production Readiness Review 


* Guidance, not statute 


Changes in next update to Defense Acquisition Guidebook 
— SE and T&E Chapters 

Incorporate TM in recommended formats of 

— SEP, TES, TEMP 

Increase TM emphasis in OSD Oversight 

— PSRs, AOTRs 
Add emphasis on TM to DAU SE and T&E curriculum 

— CLM on TM planning 


Publicize renewed TM emphasis to DoD, Service, and 
Industry organizations 


Back-up 


emphasis 


TRL 1 


Component / 
Technology Maturity 


MSA 


MS B 


TRL 6 
EMRL 2 


CDR 


EMRL 3 


time 


Engineering 


MS C 


TRL 7 
EMRL 4 


System / 
Technical Maturity 


Decision: 
CD* 1. Basic principles observed and reported 
. Technology concept and/or application formulated 

3. Analytical and experimental critical function and/or 

characteristic proof of concept 
MS A* 4. Component and/or breadboard validation in a laboratory 

environment 

5. Component and/or breadboard validation in a relevant 
sailed ue 


mMsc* 7. System ae senate in an operational 
environment 
8. Actual system completed and qualified through test and 
demonstration 
9. Actual system proven through successful mission operations 


* Guidance, not statute 


Critical Technology “Off-Ramps” 


Ast Qtr FY#L 


Component 
# 1 


Component 
# 2 


Component 
#3 


Delivery P , 
ost- 

“| Shakedown 
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Ist Qtr FY#2. Ast Qtr FY#3 


TRL3. TRL4 ~~ TRLS TRL 6 
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i r Material Required in Yard 
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» Material Required 
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of ramp fallback to 
legacy ade 


TRL 6 TRL 7 TRL8 
r+)  & Material Required — 
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Lore vamp fallback to 
legacy system 


SRR: 

-All critical technologies (or alternatives) 
2 TRL-6 

-Acceptable performance, balanced vs 
capability need, cost, schedule, etc. 

-Support MDA cert at MS B 

Ajram Initiation /c\ 


-ID immature critical technologies 
w/ maturation plans 

-ID mature alternatives w/ 
performance/other trades 


Technical Review 
Exit Criteria 


lOc FOC 


Bystem Development & Demonstration Preduction & Deployment (perations & Support 


Brteokys 
Pioctepmoat Fine 


-Demo/Assess performance 
-Verify TRLs 

-Track TM progress vs plan 
-Take Off-ramp (if req'd) 


-ID immature critical 
technologies 


-ID mature alternatives 


-Assess performance w/ 
critical technologies 


-Assess performance w/ 
mature alternatives 


DT&E Role, 
“in Preps for 
Technical Reviews 
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Forward Looking Statements 


Certain statements made in this presentation that are not based on historical 


information are forward-looking statements which are made pursuant to the safe 
harbor provisions of the Private Securities Litigation Reform Act of 1995. 


These statements are neither promises nor quarantees, but are subject to a variety 
of risks and uncertainties, many of which are beyond our control, which could 
cause actual results to differ materially from those contemplated in these forward- 
looking statements. 


Existing and prospective investors are cautioned not to place undue reliance on 
these forward-looking statements, which speak only as of the date hereof. iRobot 
Corporation undertakes no obligation to update or revise the information contained 
in this presentation, whether as a result of new information, future events or 
circumstances or otherwise. 


For additional disclosure regarding these and other risks faced by iRobot 
Corporation, see the disclosure contained in our public filings with the Securities 
and Exchange Commission. 


PREDICTING THE FUTURE CAN BE DIFFICULT 


|, ROBOT THE MOVIE 


The Checkerboard “Illusion” 


same 


Here, the human visual 
system infers the color 
of the checks in the 
world, not the gray value 
in the image. 


The “illusion” reflects 
the successful design of 
the visual system, not a 
quirky failure. 


Disruptive Technologies Timeline 
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Life before and after a disruptive technology is fundamentally different. 


Disruptive Technologies Timeline 


Computers 1978 


Locked away from public 
= too dangerous for computers 


Used inside large companies 
Operational use in military 


First few “home” computers 
= inthe form of games 


Computer hacking clubs 
How-to-build-your-own books 


Undergraduate majors 
appearing 


First mass market “serious” 
home computer attempts 
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Robots 2002 


e Locked away from public 
* too dangerous for people 


e Used in manufacturing plants 
e Operational use in military 


e First few “home” robots 
= inthe form of toys 


e Robot hacking clubs 
e How-to-build-your-own books 


e Undergraduate majors 
appearing 


e First mass market “serious” 
home robot attempts 


—————— jpapor 


iRobot 


PAcKBOoT. 


Roomba 


iRobot 


ir~tobot What We Do 


PackBot wi 
Recon kit, 
Afghanistan, 
2003 


PackBot w/Recon . iRobot SUGV for . : 
PackBot : iRobot Warrior 
i PackBot w/iEOD kit 
kit w/Advanced FCS Exp. 1.1, X700 
Cb Recon kit WSMR 2007 IR 7 


iRobot is integral to the 215 Century Military 


e 81 robots per combat brigade team 


- S | (SV + 3,600 robots 


on omy ee e Spares, upgrades, training, service, and 
~~ Ss . 
Gn wie AyD hy support throughout the lifecycle 


¢ A 30+ year business opportunity 


e : 1 
Fully Integrated into: N (Soe 
a Logistical Supply Chain .\ One jeam-Ine Army/DerenseMndusty 
= Communications — 
Network 


« Training Programs 
«= Doctrine Development 
Programs 


ROBOTS IN FUTURE URBAN ENVIRONMENTS 


Air Assault Expeditionary Force (AAEF) 
Experiment, Ft. Benning, GA 2005 & 2006 


e 2005 


First question: "Captain, of all the new technologies and 
capabilities you've used during this AAEF experiment, which 
one single piece would you deploy today?" 


Answer: "Sir, the Packbot (‘SUGV’.)” 
e 2006 


First question: "Captain - what three systems employed at 
this experiment would you want to take to war?" 

"Sir, the Raven & Buster uav's and a ugv - sir, the Packbot 
(‘SUGV')." 

"Why did you single out those three?" 

"Sir, | wanted sa (situational awareness) - from the air and 
from the ground. The packbot was RELIABLE and light 
enough (to carry.) the Raven was also reliable. | said two 
uav's because | always wanted one." 


iROBOT WARRIOR 


iRobot — John Deere Robotic Gator 
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PackBot: Digital Modular Architecture 


EOD: Explosive Ordnance Disposal 


FIDO: Explosives sniffer 


Explorer: Recon & Surveillance 


Scout: Recon & Surveillance 


PackBot: Modular Chassis 


= iRobot 


iRobot PackBot - A Digital Machine... 


A 


Modular Payload Architecture 
e 1 Front Payload 
e 4 Side Payloads 
e 3 Rear Payloads 


Signals 
e Ethernet 
e USB 
e Motion Control 
e Power 
e 2 Video Channels 


Digital Architecture & 
Systems Integration 
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Technology Core Competencies 
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Growing . 
atent Portfolio: 
} atents 
More than 
Dynamic 
Sensing 


WMO, 


Nomadics FIDO + iRobot PackBot 2 
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SAFETY FROM STANDOFF: 
PackBot carries the sensor to THE DETECTION OPPORTUNITY: 


the vehicles, so Soldiers Explosives leave invisible traces of 
aren’t exposed. vapors which FIDO can sniff and detect. 


“FIDO was able to detect explosives 80 feet away. This allowed separation of the 
Soldiers and dog from the bomb, thus saving lives... .” 
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Military Engagement — The Word is a More Dangerous Place 


e High tempo military operations and 
a dramatically lower tolerance for 
causalities and POW’s. 


e U.S. volunteer force resources are 
being stretched 


Need for Operations 


Ability to support them 
We have over 350,000 SOLDIERS 


overseas in 120 countries 


e The US cannot afford to spend 
our way out of this problem 
using current technology and 
doctrine 
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Long Term Driver 


¢ Our current ability to care for the 
elderly is barely adequate and in 
decline 


e A massive increase in the number 
of elderly people is imminent 


Number of Elderly 


Ability to Care for them mia 
65 and Over Share of Population 
[__] 5-10% [I] 10-15% NN 15-20% MN 20-25% 


HE 25-30% 


e We cannot simply spend our Source: US Census, A Series 


a to a solution. 
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Europe - 2050 
(Population: 401 million) : 
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The Robot Industry 
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Trade Space 


Modularity 
Development Cost 
Survivability 


Mobility 


Weight 


Average Unit 


Snel: Production Price 
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SOUTHWEST RESEARCH INSTITUTE 


NATIONAL INSTITUTE OF STANDARDS 
AND TECHNOLOGY 


Early Decisions Affect Life Cycle Cost 


LIFE CYCLE COST 


CUMULATIVE 


LIFE CYCLE COST EFFECTIVELY 
RENDERED UNCHANGEABLE 
FOR A GIVEN DESIGNY, 


AS 


LIFE CYCLE COST 
ACTUALLY EXPENDED: 


OUT OF SERVICE 
MILESTONES 


SYSTEM LIFE CYCLE 


Soldiers will find more missions for robots than expected 


Pe 


Ke SEE THE WORLD THROUGH THE 
SOLDIERS’ EYES, AND MEASURE 
OUR SUCCESS BY THEIR SUCCESS. 


Computers 1978 


Locked away from public 
= too dangerous for computers 


Used inside large companies 
Operational use in military 


First few “home” computers 
= inthe form of games 


Computer hacking clubs 
How-to-build-your-own books 


Undergraduate majors 
appearing 


First mass market “serious” 
home computer attempts 
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Robots 2001 


e Locked away from public 
* too dangerous for people 


e Used in manufacturing plants 
e Operational use in military 


e First few “home” robots 
= inthe form of toys 


e Robot hacking clubs 
e How-to-build-your-own books 


e Undergraduate majors 
appearing 


e First mass market “serious” 
home robot attempts 
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Big Application - Manufacturing 


¢ currently: robot arms for “fixed automation” 


e future: flexible low-cost manufacture 


* dexterous assembly/fabrication of small low cost 
products 


" currently we outsource this to low cost developing- 
world labor 


» what are the technical challenges? 
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Consequences 


Completely change the world’s labor markets 
from the way they have developed over the last 
50 years 

» change the need for low-cost labor migration 

» change the face of out sourcing 


= significantly impact the labor requirements for elder- 
care in societies with changing demographics 


* CHANGE THE WAY THE MILITARY OPERATES 


Potential to create an economic tsunami that 
rivals or surpasses the silicon valley experience 


Deliver Great Product 


moot 


Make Money 


Have Fun 
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A Better Way 


More than 500 PackBot® 
Military Robots delivered 
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“You have saved 
lives today!” 


“When a robot dies 
you don’t have to write 
a letter to its mother.” 


iRobot 


THANK YOU! 
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jdyer@irobot.com 
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Big Application - Manufacturing 


¢ currently: robot arms for “fixed 
automation” 


e future: flexible low-cost manufacture 


" dexterous assembly/fabrication of small low 
cost products 


* currently we outsource this to low cost 
developing-world labor 


» what are the technical challenges? 
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iRobot Today 


142 Million in 2005 Revenue 
400 Employees* 


Offices in 3 US locations + 
Hong Kong and India 


Over the past 3 years, 
shipped over $.25 Billion of 
robots to a diverse set of 
customers 


*includes consultants and temporary employees 


Behavior-Based Robots 


e Fast connections between 
sensors and actuators Creature-like 
* control system 


e Composable behaviors 


e Multiple simultaneous 
goals with dynamic 
arbitration 


e Dynamically variable > 
degrees of autonomy > Petautt Behavior 


Attention Behavior 
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gm Default Behavior 


Our Innovation Engine 


7% 
IR&D 2 
S 
3 
oo = Innovation 3 
Gov't Ves 5 nnovation E 
R&D S = 
. i rw 
y “ENC. gor” 
15% I 
Prior 4 Quarter Investments 
as a % of Revenue Nii; 
itary Robo? 


T 
© <<< DOE 


FUTURE COMBAT SYSTEMS 
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One Team-The Army/Defense/Industry 


Manned Systems 


Unmanned Air Vehicles 
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Class IV 


tery Class III 
Infantry Carrier Command and 


Vehicle Control Vehicle 


Unattended Munitions 


NLOS LS 
* Intelligent 


Munitions 
° Unattended Ground Systems 
Mounted rT tC~™Y Sensors 
Combat System 7 
" Recon and ba — 
Surveillance Vehicle y) “é 


Unmanned Ground Vehicles 
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ARV Aslt 


Non-Line ; . 
of Sight Cannon Non-Line of Sight Mortar 


Small 
(Manpackable) UGV 


Medical Treatment 
QD and Evacuation MULE MULE 


ARV-A (L) (Countermine) (Transport) 


ARV RSTA 


FCS Recovery and 7 ? 
Maintenance Vehicle 


Armed Robotic Vehicle 
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iRobot PackBot - A Digital Machine... 
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Modular Payload Architecture 
e 1 Front Payload 
e 4 Side Payloads 
e 3 Rear Payloads 


Signals 
e Ethernet 
e USB 
e Motion Control 
e Power 
e 2 Video Channels 


IROBOT G&IR Division Sensor & 


Payload Strategy 


We shall Shop the Market for Best in Class 


We will Integrate and Market 3 Party equipment on 
Our Robots 


While We View Our Platforms as “Tightly Integrated,” 
We are Open to Licensing Arrangements and 
Partnerships 


We will Selectively Develop Sensors & Payloads when 
Customers, Technology, and/or Packaging Demands 


sponsored by: 
The Technical Support Working Group 


Summary 


$40)'B 


potential market 
Opportunity in 


2010 
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Revenue Growth 
Product Lifecycle Revenue 


Products 
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Gov. Funded R&D 33 patents Multi-Disciplinary Robotic Partners 


50 pendin Engineering Team 
i - Outsourced Manufacturing ae JOHN DEERE 
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Four Fundamental New Capabilities 


e |. The object recognition capabilities of a two year 
old child 


e Il. The manual dexterity of a six year old child 


e Ill. The ability to move around freely and to work in 
built-for-human environments 


e IV. Intuitive human interfaces 
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Big Application - Agriculture 


¢ currently: roboticizing large agricultural machines 


e future: maintenance of individual plants 
" pruning, picking, etc. 


" currently Europe and US import low cost labor, Japan has 
higher cost agriculture 


» what are the technical challenges? 
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Big Application - Elder Care 


e currently: no automation 


e future: robotic assistants 
"in-house care and nursing care 


* currently Europe and US import low cost 
labor, Japan is facing immediate challenges 


» what are the technical challenges? 
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Exploring the Great Pyramid of Giza 


Copynght - ftudciph Gatentrnk, used wih permeson 


Computers 1978 = Robots 2001 


Locked away from public 
= too dangerous for computers 


Used inside large companies 
Operational use in military 


First few “home” computers 
= inthe form of games 


Computer hacking clubs 
How-to-build-your-own books 
Undergraduate majors appearing 


First mass market “serious” home 
computer attempts 


T 
CD 


Locked away from public 
* too dangerous for people 


Used in manufacturing plants 
Operational use in military 


First few “home” robots 
= in the form of toys 


Robot hacking clubs 
How-to-build-your-own books 
Undergraduate majors appearing 


First mass market “serious” home robot 
attempts 


Four Fundamental New Capabilities 


|. The object recognition capabilities of a two year old child 


e Il. The manual dexterity of a six year old child 


Ill. The ability to move around freely and to work in built-for- 
human environments 


e IV. Intuitive human interfaces 
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Consequences 


Se 


Completely change the world’s labor markets from the way they 

have developed over the last 50 years 

= change the need for low-cost labor migration 

= change the need for out sourcing low-cost manufacture 

" significantly impact the labor requirements for elder-care in societies with 
changing demographics 

» CHANGE THE WAY THE MILITARY OPERATES 


Create an economic tsunami that rivals or surpasses the silicon 
valley experience 


Big Application - Agriculture 


¢ currently: roboticizing large agricultural machines 


e future: maintenance of individual plants 
" pruning, picking, etc. 


" currently Europe and US import low cost labor, Japan has 
higher cost agriculture 


» what are the technical challenges? 
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Big Application - Elder Care 


¢ currently: no automation 


e future: robotic assistants 
» In-house care and nursing care 


* currently Europe and US import low cost labor, 
Japan is facing immediate challenges 


» what are the technical challenges? 
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Local Ambiguity: Quadrants Same 
Information must be propagated across space. 


Local Ambiguity: Quadrants Same | 
Information must be propagated across space. 


The Checkerboard “Illusion” 


same 


Here, the human visual 
system infers the color 
of the checks in the 
world, not the gray value 
in the image. 


The “illusion” reflects 
the successful design of 
the visual system, not a 
quirky failure. 


Local Ambiguity: Quadrants Same | 
Information must be propagated across space. 


There Is a Better Way 
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VACUUMING ROBOT 


¢ Introduced in 2002 

¢ 2 million units sold 

¢ Strongly patented 

° 1% market penetration 


ireobotf 
Sca ba 
FLOOR WASHING ROBOT 

e Wet scrubs hard surfaces 


e 4-stage cleaning process 
e Cleans better than mopping 


e Recent Awards 


» Popular Science 
Best of What’s New 


= Consumer Electronics Show 
S| 
Innovations award 


« Time Magazine Top Inventions 


iRobot Today 


¢ 142 Million in 2005 Revenue ites a 

- 400 Employees* We \ iV EL eas ——T 

¢ Offices in 3 US locations + 2 ee | [At 
Hong Kong and India 


e Over the past 3 years, 
shipped over $.25 Billion of 
robots to a diverse set of 
customers 


Mysore, India Hong Kong, PRC 


2 Loui i CA, USA Washington DC, USA ; 
Sano eave . *includes consultants and temporary employees 
O a — FRODOE 


Explosive Revenue Growth 


Technology Robot Robot 
Development Development Revenue 


$150 5 
$130 4 
$110 4 


$90 4 
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Dollars in Millions 
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$30 + 


$10 +4 


1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 YTD 


iRobot 


Market Opportunity 


Sales (in Millions USD) 


iRobot Today 


Robot Sales 


Market in 5 years 


Source: Future Horizons 


iRobot 
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Sergeant tnuct, right, talke to a chopkeeper about who might have planted mines ’ 
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Behavior-Based Robots 


e Fast connections between 
sensors and actuators Creature-like 
* control system 


e Composable behaviors 


e Multiple simultaneous 
goals with dynamic 
arbitration 


e Dynamically variable > 
degrees of autonomy > Petautt Behavior 


Attention Behavior 
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gm Default Behavior 


PackBot #129 


Killed In Action 


Supportability & 
Transformation 


Good sustainability requires good 
supportability 


Bonding... 
¢ Tested musical theory that Vince Gill was the lead 


singer for Pure Prairie League on the hit “Amie” 


- They held extensive auditions that resulted in the hiring 
of (born April 12, 1957, in Norman, OK) as lead 
Singer and guitarist, followed by reeds player Patrick Bolin 


- http://www.answers.com/topic/ pure- prairie- league 
e But - learning that | must evaluate, | did further research 


- "Amie" charted in March 1975 and became a Top 40 hit. 
Of course, the song had been written and sung by Craig 
Fuller 


- http://www.answers.com/Pure% 20prairie% 20League 
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hy are We Here 


¢ Conference theme of Sustainability 


- “Test and Evaluation in Support of Operational 
Suitability, Effectiveness and Sustainment of 
Deployed Systems 


¢ Combined NDIA Committee sponsorship of T&E, 
Systems Engineers, and Logisticians 


- T&E Division tin cooperation with... 
- Systems Engineering and Logistics Divisions... 


¢e Will the logisticians in the room please stand up 


New Requirements 


DEPUTY UNDER SECRETARY OF DEFENSE FOR 
LOGISTICS AND MATERIEL READINESS 
3500 DEFENSE PENTAGON 
WASHINGTON, DC 20301-3500 


MAR 1 0 2007 


MEMORANDUM FOR UNDER SECRETARIES OF THE MILITARY 
DEPARTMENTS 


SUBJECT: Life Cycle Sustainment Outcome Metrics 


In July 2006, the Joint Requirements Oversight Council (JROC) 
established a mandatory warfighter Materiel Readiness/Sustainment Key 
Performance Parameter (KPP) (Materiel Availability) and identified Material 
Reliability and Ownership Cost as related Key System Attributes (KSAs) for 
- New acquisitions. Specific definitions of these metrics, as they will appear in the 
revised Chairman of the Joint Chiefs of Staff Manual (CJCSM) 3170.01C, 
scheduled for issuance in 1Qtr CY2007, are contained in Attachment 1. 
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ew Metrics 


¢ Life Cycle Sustainment Outcome Metrics 
- Material Availability (KPP) 
- Material Reliability (KSA) 
- Ownership Cost (KSA) 


¢ Goals developed for each program early in 
the Concept Development process 


Conference Take Aways 


¢ Meeting sustainability standards is 
achievable but not free or easy 


- Lack of robust systems engineering culture 
- Cost drivers force tough decisions 


- Lack of understanding the linkage between 
events regardless of contract phase or funding 
profile 


234 Annual National Test and 
Evaluation Conference 


Encroachment Issues 
Impacting the 
Major Range and Test Facility Base 
(MRTFB) 


Mr. Bill Egan 
Institute for Defense Analysis 


March 13, 2007 


The MRTFB 


The core set of Test &Evaluation (T&E) ranges & 
facilities within DoD 


Represents a broad base of critical T&E capabilities 


A national asset, sized, operated, and maintained 
primarily for DoD T&E 


Available to other users (i.e., DoD, federal agencies, 
and allies within security provisions) 


Operated under uniform funding guidelines 


Governed by DoD Directive 3200.11 


Major Range and Test Facilities Base 


., és 22 Sites: Army-7; Navy-6; Air Force-7; Defense Agency-2 


Ss 
Sen, MENT OF pore 


~ Test and Evaluation Management Center, 
Nevada Test & Training Range, NV includes the National Capitol Region 
} Utah Test & Training Range, UT 
/ Zhe 


Dugway Proving Ground, UT 


White Sands Missile Range, NM 
High Energy Laser Systems Test Facility, NM 


& 


Naval Undersea Warfare Center, 
Nanoose/Dabob Range, 
Keyport, WA 


Naval Air Warfare Center 
China Lake, CA 
30" Space Wing, 
Vandenberg AFB, CA 


Naval Air Warfare Center, ae 


\ a 
\ Aberdeen Test Center, MD 
1 


Naval Air Warfare Center, 
Pax River, MD 
Arnold Engineering Development 
; . Center, TN 
Electronic Proving Ground, AZ ie 


Point Mugu, CA 


Air Force Flight Test Center 
Edwards AFB, CA 


Yuma Proving Ground, AZ 
(Includes Cold Region Test Ctr, AK & 
Tropic Region Test Ctr, Hl/Puerto Rico 


45th Space Wing, FL 


ba 
Air Armament Center, FL Atlantic Undersea 


> Joint Interoperability 
Test Command, AZ 
(Includes 46 Test Group Test & Evaluation Center, 
Reagan Test Site at Holloman AFB, NM) FL / Andros Island 
Kwajalein Atoll 


Pacific Missile Range Facility, Hl 


Legend: 
Army, Navy, AF, Defense Agency 


MRTFB Survey Questions on 
Encroachment 


1. Describe the most significant encroachment issues impacting 
your MRTFB 


2. Describe any potential candidates for partnering projects in 
the DoD Conservation Buffer Program using DoD’s authority 
in 10 USC Section 2684(a)? 


3. Describe any local community or government partnerships 
you are actively engaged in to help resolve or mitigate your 
encroachment issues. 


4. Provide any notable examples of successful proactive 
engagement in avoiding or overcoming a encroachment 
impacts. 
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1. Describe the most significant 
encroachment issues 
impacting your MRTFB. 


MRTFB Encroachment 
Responses to Survey 


ES a SS a a a a a a a 
SS A A OG 
a oe ee 
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INaweck [| x | x | x |x |x |x |x | | | 
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Air Force x 
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Fa a Se eS (ae SS a a 
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aan] [seen |SacneeT (NE) 
ENV=Environmental* AS = Air Space RED = Most Significant 
FS = Frequency Spectrum RL = Range Limitations Encroachment 
US = Urban sprawl SR = Sea Range Limitations FS= 8 
ENG = Energy UDAs = Undocumented Aliens USs= 6 
NC = Natural & Cultural ENV = 2 


* Environment = anomaly data point 


Reported Sources 
of Encroachment 


@ Air Force 


Two Major 
Encroachments 


e Frequency Spectrum 


e Urban Sprawl 


Increasing Demands for Radio 
Frequency (RF) Spectrum 


Increasing demand for bandwidth and decreasing available Specuum: RF interference 
from/with commercial and unlicensed users. 


In 1986, the data rate for the F-15 was 7 
256 Kbps In 1997, the data rate for the F-22 was 
5 Mbps 


Spectrum Reallocated 


TOTALS 
a 58 MHz 
f TSPI 40 MHz 
100 MHz 
> LBand 99 MHz 

) ™ 


ong 100 MHz 
0 


Oo Spectrum iru Allp pcation Prior» WRC 92 Total Lost = 93 MHz 


Frequency Spectrum Example 


e 22 Examples Reported 


¢ 15 MRTFB’s Responded with similar issue. 


“Our crucial electromagnetic spectrum assets are 
continuously threatened with reallocation to non- 
Government usage to bring in auction profits to the 
US Treasury. The MRTFB frequency bands that 
remain are being encroached upon by billions of 
unlicensed/low-powered commercial Part 15 
Devices” 
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Development of adjacent private and public lands, and commercial _ 
developments such as condominiums, air fields and recreation facilities 


oD Ranges 
Air Force 


a Army 
HE Marine Corps 
MM Navy 


SUAS 
Pop. per Sq Mile 
0-4 
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Urban Sprawl Examples 


25 Examples Reported 


NAWCWD-CL: “Urban growth within the R-2508 Complex and 
within the boundaries of IR-200, could also negatively impact 
the NAWCWD-CL mission. 


46th TW/TG Condominium development along the Gulf coast if 
the height impacts instrumentation line-of-sight over the Gulf of 
Mexico. 


NTTC Nellis AFB’s ability to perform its mission is endangered 
by the ever increasing pressure of urban encroachment. 
Development can threaten or have the potential to threaten 
current and forecasted Air Force mission requirements at Nellis 
AFB (NAFB), Creech AFB (CAFB) and the Nevada Test and 
Training Range (NTTR) if the type of development is non 
compatible with the military mission. 
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2. Describe any potential candidates for 
partnering projects in the DoD 
Conservation Buffer Program using 
DoD’s authority in 10 USC Section 
2684(a)? 


Buffer Partnership 
Examples 


14 Examples Reported 


AFFTC We have a significant candidate project for partnering 
under the provisions of 10 USC Section 2684(a) -— The R-2508 
Sustainability Plan. The focus of this plan is to protect the 
most critical operating areas within the R-2508 from a joint (tri- 
service) perspective. Currently three sub-projects have been 
prepared within the R-2508 Sustainability Plan. 


APG is currently partnered with Harford County Land Trust for 
the Army Compatible Use Buffer (ACUB). Land has been 
secured surrounding the DOD’s only hilly, cross-country 
automotive test track which has been in operation for over 60 
years. 
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Survey Question Three 


3. Describe any local community or 
government partnerships you are 
actively engaged in to help resolve or 
mitigate your encroachment issues. 
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Community Partnership 


Examples 
¢ 57 Examples Reported 


¢ 30th SW agreements with various organizations for 
the evacuation, sheltering, or closure of local public 
and private population centers surrounding 
Vandenberg AFB, 


¢ YPG west of the YPG Laguna/Cibola Area with the 
State land near Martinez Lake on the Colorado River- 
-attempts in the past of private citizens attempting to 
purchase State land for putting in an airfield within 
restricted airspace. YPG worked closely with the 
State to disapprove this action. 
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@ ) Survey Question Four 


4. Provide any notable examples of 
successful proactive engagement in 
avoiding or overcoming 
encroachment impacts. 
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Proactive Engagement 
Examples 


¢ 68 Examples Reported 


¢ AEDC There is a very active Arnold 
Community Council that provides an avenue 
to address any encroachment issues. 


¢ AFFTC partnership with the wind industry to 
help guide development away from our most 
critical operating areas. 

¢ APG has an active Noise Monitoring 
program. Monitors are in place in a 4-county 
area around Aberdeen Proving Ground and 
are constantly monitored. 
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« Passive Involvement 


« Proactive Involvement 
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~~ | In Trouble?? 
c= (Passive Involvement) 


Act Like You Know What You Are Doing and Keep Going 
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Proactive Involvement 


e Addressing Frequency Spectrum Encroachment 

— Central T&E Investment Program (CTEIP) investments 
¢ Spectral Efficiency Technologies 

— Range Spectrum Requirements Working Group (RSRWG) 

— World Radiocommunication Conference (WRC) 2007 Agenda Item 1.5 
¢ Competing globally for additional spectrum 

— DoD Spectrum Office, ASD(NII) 
¢ Developing strategic plan for spectrum 


e Addressing Urban Sprawl 
— FY2003 National Defense Authorization Act (NDAA) - Authority for Military 
Departments to partner with state and local governments and not-for- 
profit land trust associations to acquire easements and other interests in 
lands adjacent or in the vicinity of military installations. 
— DoD Readiness and. Environmental Protection Initiative (REPI) 


¢ DoD Conservation Buffer Program 
Per 10 USC Section 2684(a) 


— Very Successful Program 
— Need more Test Range Participation 
21. 
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RRPI Successes to Date 


e Five of eight RRPI provisions have been enacted: 


MBTA Military Exemption 

MMPA Harassment 

ESA Critical Habitat 
Conservation Partnering (buffers) 
Surplus Land Disposal 


¢ Benefits to DoD are tangible 


MBTA —- Permanent USFWS rule codifying exemption now in place 
MMPA - Has increased some exercise and training flexibility at sea 
(but other challenges remain) 

ESA Critical Habitat - Ranges now preserving endangered species 
under terms of their INRMPs instead of with newly declared critical 
habitat 

Conservation Partnering / Buffer program has leveraged DoD funding 
and established win-win strategic partnerships. Significant acres of 
land already under 2684a conservation to DoD benefit. 
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MRTFB Survey Responses 


¢  Environmental—17 - Air Space - 10 


— Army-4 
— Army-3 
— Navy -6 — ap a” 
— Air Force - 7 — Air Force -5 


« Frequency Spectrum—15 


— Army -4 ¢ Energy Projects - 9 


— Army-0O 
= Navy SS = ie -3 
— Air Force -5 ; 
_  sTIc-1 —  Atr Force -6 


¢ Urban Sprawl—14 ¢ Sea Space Limits - 6 


— Army-0O 
Sa = isa -4 
— Navy4 7 
— Air Force -6 Air Force - 2 


® Natural & Cultural Resources -13 * Undocumented Aliens - 4 


— Army-3 
— Army-3 7 eae : 
— Navy-5 
— Air Force-5 — Air Force -1 


e Range Limitations - 11 
— Army-2 
— Navy-4 
— Air Force-5 


Increasing Footprint to Support T&E 


Increasing three dimensional foot print requirements for evolving faster, longer range 
weapon systems; environmental concerns and urban development initiatives that 
restrict airspace use 


Land Attack Cruise Missile 
otional) 


CROSSRANGE (nmi) 


0 20 40 60 80 100 120 140 160 180 
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Off-shore drilling platforms, wind energy wind mills, Liquid Natural 
Gas (LNG) transport and storage, mandated energy corridors 


Proposed Energy Corridors; initially 11 Western States; Soon National Footprint 
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Increasing Demands to Protect 
Natural and Cultural Resources 


Existing and newly established national monuments, water recreation sites, 
parks, forests, sacred lands and archeological sites 


Examples 


30th SW: Point Conception Lighthouse — Acquisition of a decommissioned lighthouse 
(30 acre parcel) from the U.S. Coast Guard will prevent private ownership of the 
property, where the AF operates a radar system in support of MRTFB operations. 


AFFTC: Supersonic and low-level operations can generate pressures from the public to 
curb operations or conduct them in other areas. An example of this is the operation of 
parks and wilderness areas. 


NAWCWD-CL China Lake: There are literally thousands of archeological sites on the 
ranges at China Lake. They require substantial management effort and financial 
support, primarily for surveys. 


NAWCWD-PM: California Coastal National Monument - The California Coastal National 
Monument was created in January 2001. It designates all the rocks along the California 
shoreline as a national monument under the Bureau of Land Management (BLM). The 
rocks around San Nicolas Island are included. 
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Increasing Demands on Air Space 


Los Angeles Air Traffic Control Center has reported record traffic counts 
over the last year. The R-2508 airspace is surrounded by three of the top 
ten busiest airports in the country; San Francisco bay area to the north, Las 
Vegas to the east, and Los Angeles basin to the south. 


Examples 


AFFTC: Airspace is a sought-after commodity, from other DOD Services, 
commercial and private sources. 


NAWCWD-CL: Initial planning for a new airport to serve Las Vegas is underway. 


UTTR: Overflights, deviation requests and denial of R-6404D at required altitudes 
continues to be a challenge to ensure UTTR MRTFB mission accomplishment 


EPG: The FAA has ruled against UAV flight outside of restricted airspace. 


NAWCAD-PR: Pressure from increased private and commercial flights (Eastern 
Shore and SUA Spill-in) 28 


Increased commercial demand for airspace, long haul and terminal, at all altitudes; 
Results in restrictions on DoD Special Use Air Space 


Tracks of Civil Take Offs 
November 22, 2001 


Take Off Time (EST) Special Use Airspace 
210.2200 
160.1700 
1400-1500 
4100-1200 
0900-1000 [Restricted 
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Increasing Demands on Range 
Space 


Examples: 


NAWCAD-PR: Reduced access to land based targets and surface operating areas at the 
Bloodsworth Island Range results in a loss of potential capabilities and benefits. 


UTTR: Large footprint weapons require additional property be considered when preventing 
physical encroachment. 


YPG: Cold Regions Test Center’s (CRTC’s) largest encroachment comes from other Army 
units. US Army, Alaska (USARAK) has increased in size with the drawdown of OCONUS 
units. With this increase in size came an increase in the amount of training conducted in the 
same areas as CRTC uses for testing. 


NAWCWD-CL: Urban growth within the R-2508 Complex and within the boundaries of IR-200, 
could also negatively impact the NAWCWD-CL mission. (IR-200 is the corridor used for 
testing cruise missiles fired from the Sea Range to the China Lake Land Ranges and is the 
only such corridor on the West Coast.) 


NAWCWD-PM: Channel Islands National Marine Sanctuary (CINMS). The Sea Range 
encompasses the CINMS, one of fourteen national marine sanctuaries managed by NOAA. As 
part of the CINMS Management Plan update, NOAA has proposed expansion of the sanctuary 
along with changes to the regulations that address military activities. 30 


Increasing T&E Safety & Security 
Concerns from Trespassing Aliens 


Undocumented Aliens (UDA’s)— Danger to undocumented Immigrants 
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Low-Cost Epoch-by-Epoch™ Network-Centric 
Positioning Unit (ENPU) for FCS Testing 


23'¢ Annual National Test & Evaluation Conference 
Hilton Head Island, SC 
March 12-15, 2007 


Presented by: 
Dr. Jeff Fayman 


About Geodetics 


Geodetics, Inc. is a Woman Owned Small Business 
(WOSB) based in San Diego, California. 


Geodetics specializes in high-precision, real-time 
position and navigation solutions based on its 
proprietary Epoch-by-Epoch™ technology. 


Geodetics offers a full range of GPS based products 
and services. 


Real-Time Dynamic (RTD) Technology 


RTK Reference Geodetics 
Differential/RTK Networks RTD 


Stand Alone 


15m 3m 1m 2cm 2cm 2cm at 50 km 
Baseline Baseline < 1m at 100’s km 
10-15 km 50 km 


Geodetics is Introducing Real-Time GPS Reference Network Technology to the 
Army at Ft. Bliss and Ft. Hood. And the Navy at SPAWAR. 


This Technology has been Deployed in many Civilian Applications World-Wide 
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Geodetics Total Solution (Civilian) 
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Real-Time, High-Accuracy 
Network-Centric Positioning for Mobile Force 


Epoch-by-Epoch™ 
technology embedded in 
rovers offers real-time 
centimeter level relative 
and absolute positioning 
of many players. 


The reference network 
enables extended range 
operations of the players 
and centralized 
command and control ee * 
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Epoch-by-Epoch™ Network-Centric 
Positioning Unit (ENPU) 
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ENPU Accuracy Testing 
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ENPU Accuracy Testing (cont.) 


Outdoor Environment — Open Sky 


Indoor Environment 


Note: Testing for GPS only capabilities i.e. no IMU, no smoothing etc. 
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ENPU Accuracy Testing (cont.) 


STAND-ALONE PERFORMANCE 


Test Date North StdDev East StdDev Up StdDev 
4/30/2006 4.55727 3.30303 9.56708 66399 solutions with 1511 outliers (1.7%) (0 non-solutions) 


Test Date North StdDev East StdDev Up StdDev 
5/29/2006 6.92149 4.53512 14.18528 84961 solutions with 1101 outliers (1.3% 


DIFFERENTIAL PERFORMANCE 


Test Date Reference Baseline North StdDev East StdDev Up StdDev Solutions 

4/30/2006 ARTR 11.08218 0.6937 0.59169 1.79606 86362 solutions with 117 outliers (0.1%) (37 non-solutions) 
4/30/2006 S105 2514.32131 0.64314 0.56589 1.55919 86399 solutions with 113 outliers (0.1%) (0 non-solutions) 
4/30/2006 P472 13538.93553 0.67077 0.58248 1.62057 86399 solutions with 130 outliers (0.2%) (0 non-solutions) 


Test Date Reference Baseline North StdDev East StdDev Up StdDev Solutions 

5/29/2006 ARTR 10.92737 3.45577 2.6116 7.13446 83560 solutions with 1056 outliers ( 1.3%) (2639 non-solutions) 
5/29/2006 S105 2514.92052 3.42554 2.79797 7.0174 83897 solutions with 1082 outliers (1.3%) (2502 non-solutions) 
5/29/2006 P472 13537 00244 3.41271 2.60889 7.06108 83846 solutions with 1017 outliers (1.2%) (2553 non-solutions 
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ENPU Accuracy Testing at Ft. Bliss 


Ft. Bliss Test Site 
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ENPU Accuracy Testing at Ft. Bliss 


Test 1: Topcon at “L10” Test 4: Topcon at “L9” 
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ENPU Accuracy Testing at Ft. Bliss 


Test 2: SiRF at “L10” 


Test 3: SiRF at “L9” 
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Test 2: SiRF at “L10” 
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Test 3: SIRF at “L9” 
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Epoch-by-Epoch™ Test Verification 


° Geodetics technologies were extensively tested by DoD and are making their 
way into many projects in the Military 


¢ The Central Test & Evaluation Investment Program (CTEIP) has funded a 
program to evaluate the performance of Epoch-by-Epoch™ (EBE) 
technology for test and evaluation instrumentation applications. 


° EBE technology has undergone extensive testing in both live and simulated 
tests under strenuous environments, including: 


¢ Low Dynamics 


¢ High Dynamics 
¢ SAASM under High Dynamics 


¢ Attitude Determination 
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ITEA Award 


The preceding tests results were published in the October 2004 
International Test and Evaluation Association Journal under the title: 


"Epoch-by-EpochTM Real-Time GPS Positioning in High Dynamics and 
at Extended Ranges“ 


The paper was selected by the ITEA publications committee as the most 
important paper published in 2004. 


Geodetics, together with its co-authors received the award at the annual 
ITEA International Symposium in September 2005 
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Nunn-Perry Award 


Geodetics and Lockheed Martin entered a three year DOD sponsored Mentor-Protégé 
(MP) agreement in August, 2005. The program was awarded by the OSD Office of 
Small Business Utilization through the U.S. Navy SPAWAR, San Diego, CA, Joint 
Robotics Program (JRP). 


Under this program Lockheed Martin is mentoring Geodetics, in the manufacture of 
high-accuracy, real-time geo-location sensor systems based on GPS technology. This 
MP agreement enables Geodetics to deliver turn-key hardware/software solutions to 
the Government. 


This year, the Geodetics/Lockheed Martin Team was selected as the top Mentor- 
Protégé team out of several hundred such DOD M/P teams. 


The Geodetics/Lockheed Martin Team will receive the prestigious Nunn-Perry award 
in March recognizing the team as the best in the DOD. 
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E-2 Hawkeye / C-2 Greyhound 


¢ Very successful carrier suitability tests of the NP2000 
propeller on the E-2 Hawkeye 


— Shore based flying qualities and performance 


— Shore based catapult launch and arrested landing Stru@ilira /feactténal 
demonstration 


— Shipboard tests aboard the USS JOHN F KENNEDy A a — 
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¢ Joint Precision APPLOAeH — 
System © 


— JPALS willprovide shore and shipboard 
precision approach systems 


— Shore based system uses a Local Area 
Differential GPS (LDGPS) solution. 


— Sea based system uses a relative solution 
(Shipboard Relative GPS (SRGPS)). 


¢ Required for N-UCAS 


% ¢ Will be implemented in all fixed and rotary wing 
aircraft 4 @ 


* ATC & landing during Emissions Control (EMCON) — 
¢ Air Traffic Management for N-UCAS/UAV’s qe 
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¢ Revolutionary Design 
— Initial Fleet capability in 2013 
— Two helicopter landing spots 
¢ MH-60R 
e UAV’s 
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USS George H. W. Bush 
CVN-77 
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¢ Similar to USS Ronald Reagan, CVN-76 
Se SIcUMOMaliitcreme lit 
— Three arresting gear wires 

¢ Enters service in 2009 

¢ Modernized island 


¢ New radar tower -=2 
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M&S — General 


¢ Historical improvements 


— Catapult minimum endspeed 
predictions 


— Approach airspeed evaluations 
— Degraded flight control modes 
and emergency configurations 
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UAV/IUCAS Challenges 


¢ Performance specifications 
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Response Challenges 


* Quick response test requirements to provide 
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requirements or technical issues 


¢ Within last year, we have deployed teams to: 
— Carrier operating in WESTPAC (PALS support) 
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¢ On-aircraft test aids 
— Increased capability while testing on ships 
— Dial-a-Function 
— Net capable TM 
¢ Creating new techniques 
— JPALS data collection 
— UAV test requirements and 
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¢ Small UAV’s 
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Summary 


Large number of aircraft and ship test programs facing the 
ship suitability T&E group 
Must be able to Support.these programs with less manpower 
— Test team versatility 
— Increased dependency on M&S 
— Better data analysis tools 
Must learn from past programs 
— C-130 
— AV-8B STOVL 
=, Conventional Ski-Jump 
Develop new and innovative)test techniques 
— JPALS 
— N-UCAS 
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Background 


e Importance of reliability to ownership cost has been studied and 
documented for many years* 


* Other aspects such as maintainability, commonality, component 
packaging, and cycle time also studied, although less 
frequently* 


e Each studied more or less in isolation 


¢ Not as well understood is the relative and combined importance 
of five dimensions 
— Reliability 
— Maintainability 
— Component packaging, e.g. choice of line replaceable modules 
(LRMs) versus Line Replaceable Units (LRUs) 
— Commonality, and 
— Performance of the support process 


*References on slide 17 


Purpose of Presentation 


¢ Using data representative of a complex ground 
platform electronics subsystem, examine the relative 
importance of the five dimensions from the 
perspective of ownership cost 


¢ Show how plausible improvement in a dimension, or 
combination of dimensions, affects ownership cost 


¢ By characterizing what amounts to a trade space 
analysis can help inform the allocation of constrained 
test and evaluation resources 


Data 


* System: high technology electronics used on mobile ground platforms, with 
two types of LRUs 


— LRU A: $81K each, 5 per system, mean time between failure (MTBF) = 4,000 
hours; comprising 7 types of SRUs; 


— LRU B: $70K each, 2 per system, MTBF = 4,650 hours; comprising 9 types of 
SRUs 


¢ Equipment density: 3,645 platforms total 
* OPTEMPO: 141 hours per month per system for 20 year service period 


* Support system: two-level maintenance with nominal performance 
characteristics (e.g., 4-hour troubleshoot, remove, and repair; 45-day turn- 
around on LRU maintenance; cost of repair at sustainment level = 22% of 
production unit cost) 


* Caveats: data constructed for this analysis by analogy with systems fielded 
and in design, representative of such systems, but not data from any 
specific system 


Method 


¢ Used Logistics Support Activity (LOGSA) Cost Analysis 


Strategy Assessment (CASA) model as primary tool 


¢ Added spares holding cost manually—not addressed in 
CASA* 


¢ For LRU vs. LRM comparison, used results of recent 
study by Army Material Systems Analysis Agency 
(AMSAA) for Future Combat Systems (FCS) program to 
parametrically adjust CASA spares cost 
— CASA not well structured for such a comparison 


— AMSAA analysis used Selected Essential-item Stock for Availability 
Method-Life Cycle Cost (SESLCC) model 


— CASA and SESLCC calculated spares costs for LRU case are 
comparable, hence parametric adjustment is reasonable approach** 


*Holding costs include cost of capital, losses due to obsolescence, other inventory losses, storage costs 
**Model descriptions in backup 


Limitations on Analysis 


¢ Examined sum of classic operations and support 
(O&S) and field-level spares cost 


¢ Did not address RDT&E or production costs other 
than spares 


- Did not address sustainment (wholesale) level spares 
costs or support equipment costs 


> Treated sustainment-level repair costs as material 
costs (i.e., labor implicit in material) 


* Since time frames for all analysis are identical did not 
discount 


- Did not examine changes in unit cost, which “usually” 
decreases as failure rate decreases 


Quantification of Dimensions 


Dimension Operationalized As 


_— Failure rate Failures per 
Reliability (1/ MTBE) 1,000 hours 


Maintainability Mean time to repair (MTTR) 
Component LRU format or NA 
Packaging LRM format 


Number of distinct configurations 
Commonality of functionally equivalent Count 
systems 
Performance of Turn-around time from field to Days 
support system depot and back y 


Experimental Design 


0.4 
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180 11 
Turnaround time (days) 
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Component Packaging Results 


Support Cost 
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Support System Performance Results 
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Commonality Results 


Support Cost 
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All Worst Case and All Best Case 
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Evaluation 


(Over Range Examined, Given Characteristics of System Examined) 


¢ Order of impact 
— Reliability (2.19:1) 
— Commonality (2.07:1) 
— Support system performance (1.53:1) 
— Packaging (1.17:1) 
— Maintainability (1.03:1) 
* Worst-to-best comparison: > 5.3:1 change in 
support cost 
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Interpretation 


¢ All dimensions except maintainability strongly affect support cost 
— All except maintainability affect labor and spares 
— Maintainability affects labor 


* Comparing best combined case to worst combined case 
potential for > 5:1 change in support cost 
— Largest impact comes from design choices: reliability, commonality, 
packaging 
— Confirms notion that early design decisions “lock in” support cost 
— Since unit cost normally decreases as failure rate decreases, may 
be underestimating 
¢ Results intended to be representative of complex, high density 
electronics system on mobile ground platforms 


e Less complex, lower density systems in other environments 
likely to have similar trends but different magnitudes 
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Selected References 


i; Dwight E. Collins, “Logistic Support Cost Commitments for Life Cycle Cost 
Reduction.” LMI, June 1977. (An early but certainly not the only early look at 
reliability, maintainability, and cost) 


a Mark | Knapp and Joseph W. Stahl, “Assessment of Avionic Equipment Field 
Reliability and Maintainability as Functions of Unit Cost,” Presentation to the 
16" Annual Department of Defense Cost Analysis Symposium, October 4-7, 
1981 (Established a direct log/log relationship between unit cost and failure 
rate) 


3. James A. Forbes, Donald W. Hutcheson, and Beirn Staples. “Using 
Technology to Reduce Cost of Ownership.” LMI LG404RD4, April 1996 
(Showed the relative contribution to ownership cost of reliability and 
maintainability. Sized the potential for ownership cost return on investment in 
reliability-related technologies, compared relative return on reliability to 
maintainability) 


A. Dennis L. Zimmerman, et al., “Quantifying and Trading Off the Benefits of 
Reducing Order and Shipping Times.” LMI LG501R1, September 1997 
(Examined cost benefit of support system performance improvement) 


PAGE 17 


backup 


aA Ce mae Ks) 


Models Compared 


Cost Analysis Strategy Assessment 


(CASA) Model 


Accounting model with embedded spares 
algorithms* 

Developed by Defense Systems 
Management College 

Maintained by Army Logistics Support 
Activity 

Calculates the total cost of ownership 
including research, development, test, and 
evaluation (RDT&E); acquisition and 
production; operations and support; and 
disposal 


*CASA 2002 embedded documentation (March 2003) 


Selected Essential-item Stock for 
Availability Method (SESAME) 


Model** 
Army standard initial provisioning model. 
Optimizes the mix and placement of 
spares to achieve an end item 
operational availability (Ao) requirement 
or the maximum Ao for a dollar goal 
input 
Maintained by Army Material Systems 
Analysis Agency. 
SESAME becomes SESLCC model 
when augmented with additional logic to 
capture other support costs. 
Does not address RDT&E or production 
other than spares 


**http://findarticles.com/p/articles/mi_qa3766/is_200207/ai_n9127603 
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EXPEDITIONARY 
FIRE SUPPORT 
SYSTEM 


Ammunition 
Qualification Test 
Program 


=e Program Overview 
ge System Description 
mg lest Program 

=m Lessons Learned 


m Program Sponsor - MARCORSYSCOM, PMM-145 
General Dynamics Ordnance and Tactical Systems, Prime 
Contractor 

® Rapidly field the system with four ammunition variants 
® Internally transportable in the V-22 


gs NSWCDD and NSWCIH Tasked to Qualify Ammunition 
® NSWCDD, G31, Light Weapons Branch 
® NSWCDD, G60, Test and Evaluation Division, plan, coordinate and 
conduct test activities 
® Test program started in August 2005. 


B Qualification ammunition delivered in December 2005, major tests 
completed by November 2006 


Test Program Organization 


EFSS Mission 


EFSS is a “close supporting, all weather, quick 
responsive indirect fire system supporting 
Marine Expeditionary Units in the Ship-to- 
Objective Maneuver” providing accurate fires 
from with the 120mm rifled towed mortar to 
8.1km. The major components of the EFSS are 
the 120mm rifled towed mortar weapon, the 
weapon prime mover, a suite of 120mm rifled 
ammunition, the ammunition supply vehicle, an 
ammunition trailer, and the ballistic control 


computer. 


The EFSS will be the principal indirect fire 
support system for the vertical assault element 


of a Ship to Objective Maneuver (STOM) force. 


The EFSS will be employed within the Ground 
Combat Element (GCE) of the Marine Air 
Ground Task Force (MAGTF). 


Operated by a five-man crew, the EFSS 
provides indirect fire support that can be 
pe over friendly troops and out to ranges 
of 8.1km. 


a 


NAVSEA 


—— 
WARFARE CENTERS 
DAHLGREN 


Development, Procurement & 
Urgenc 


m Development 
& IM compliant fill for High Explosive Round 
& Modifications to the ammunition containers 
® Qualification Testing 


m Procurement 
® LRIP Decision June 2006 
® Full Rate Production August 2007 
® 1OC August 2007 


m Urgency 
& Mission Need Statement (April 2001) 
B IOC FY2007 
B® High Priority USMC Program 


XM327 120mm Rifled Towed 
Mortar 


g Original Design in Service Since 1971 
® Fielded by four NATO countries and Japan 
® Estimated 500,000 founds fired 

m= Operated By a Five-man Crew 


@ Spin Stabilized Projectile Provides Improved 
Accuracy 


Ammunition Suite 


BEFSS Will Be Deployed With Four Types Of 
Rounds 
® XM1101 High Explosive (HE) w/IM modifications 
® XM1103 Smoke (White Phosphorous (WP)) 
B® XM1105 Illumination (ILLUM) 
® XM1107 Practice (PRAC) 


Ammunition 


EXPLOSIVE 
M767A1 (PBXW-128) 
FUZE BOOSTER BODY 


BASE OF SHELL 


OBTURATING 
RING 


PROJECTILE 
WEIGHT 


DESIGNATION 


EXPLOSIVE 
FILL 


PROJECTILE 
LOT NUMBER 


BASE LOT 
NUMBER 


CHARGE HOLDER 
LOT NUMBER 


BLACK POWDER 
& 
ZENON POWDER 
CHARGES 
“GBPa 125 and GBPa 117” 


SPRING CLIPS = EJECTING 


CHARGE 
“SDIISI" 


NAVSEA 


WARFARE CENTERS 
DAHLGREN 


Ammunition 


Ammunition Trailer 


m Loaded 6X5 Horizontally for Vehicle Transport and 
Palletized 6X6 Horizontally for Shipping 
(XM1101 HE, XM1105 ILLUM, and XM1107 PRAC 
Rounds) 


m Loaded 5X5 Vertically for Vehicle Transport and Shipping 
Palletized (XM1103 SMOKE Rounds) 
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EFSS Ammunition 
ing/Storage Container 


m Rounds Packaged in Metal Ammunition Containers 
® PA117 vented and PA103 containers 


= Blowout Panels for Rapid Out-Gassing Venting 
m Relief Valve for Slow Out-Gassing Venting 
= Performance Oriented Packaging (POP) Certification 


Complete 


Container (Cont’d) 


= XM1103 Smoke Round Packaged Separately 
From Its Tail Charge to Allow for Vertical 
Palletizing and Transport 


One Warhead 
Two Tail Charges : Per Container 
Per Container 


DAHLGREN 


Stockpile to Target Sequence kee 


| | 
: en 3 Trans- ‘Storage | Transport 3 Storage | Operations l 
: : : : 
| : port 3 | ! : Trans- | Use/ : 
; | ; | 3 i port ! Disposal | 


R,A,S,H Landing Zone 
R,A,S,H 
R,A,S,H 


Demil/ 
Disposal 


d---------- 


A[VSTOM] 


R,A,S,H 


KEY CONUS Range 
A= Air 


se en Line Color 

= Highway 

CONUS = Continental United States Blue -- Weapon 
OCONUS = Outside Continental United States Gold -- Ammo 
ASP = Ammunition Storage Point Black -- Combined 


VSTOM = Vertical Ship to Objective Maneuver 


Assessment (THA 


mw THA Conducted in January 2005 


gw Threats Analyzed 
® Thermal environments produced by hydrocarbon fuel fires 
® Shock environments produced by detonation of other weapons 
oe ue environments by bullets, fragments, shaped charge jets, 

spa 

mw Determined Tests To Conduct 

Slow Cook-Off (SCO) 

Fast Cook-Off (FCO) 

Bullet Impact (Bl) 

Fragment Impact (Fl) 

Sympathetic Detonation (SD) 

Shaped Charge Jet Impact (SCJI) 


m Determined Test Not To Conduct 


& Spall Impact (SI) — not transported or stored in armored vehicle or 
armored enclosure 


TEST PROGRAM 


gm Harmonized IM and Hazard Classification 
Test Plan. 

= Comprehensive Environmental/Durability 
Test Series. 

= Collaboration Between NSWCDD, NSWCIH, 
NSWCCD And Eglin AFB for test execution. 


oa a 
—— 

WARFARE CENTERS 
DAHLGREN 


12 HE AURs 9 HE AURs 2 HE AURs 
12 Illum. AURs 9 Illum. AURs 2 Ilum. AURs 
12 Practice AURs 9 Practice AURs 2 Practice AURs 


2 HE AURs 2 HE AURs 
2 Ilum. AURs 2 Ilum. AURs 


18 HE AURs 2 HE AURs 
18 Illum. AURs 
2 Practice AURs 2 Practice AURs 18 Practice AUR 


12 Smoke Proj 6 Smoke Proj 2 Smoke Proj 


2 Smoke Proj 2 Smoke Proj 18 Smoke Proj 


12 Tail Charges i g 2 Tail Charges 2 Tail Charges 2 Tail Charges 18 Tail Charges 


LEGEND 
AUR -— All up round 


Vibration 


Tran/Ship/Helo 


V & RI- Visual and radiographic inspection 


4-Day T&H 


V&RI 


40 Foot Drop 


ENVIRONMENTAL/DURABILITY eee 


DAHLGREN 


Environmental‘ Durability Safety Tests 
45 Each Round Type 


RAIL-STD —310F 


High Temperature 
Storage 


RIIL-STD —310F 


Low Temperature 
Sterage 


MIL-STD —310F 


Low Pressure 
ltitude 


G rounds) 


MIL-STD —310F 
Temperature Shock 
© rounds) 


C13 rounds) C15 rounds) 


Fire 


CG rounds) 


Fire 
CS rounds) 


PAIL-STD —310F PAIL-STD —310F 
Secured Cargo Secured Cargo 
Vibration Vibration 
C10 rounds) 10 rounds) 


MIL-STD —310F MIL-STD —310F 


Immersion 


G rounds) 


Humidity 
G rounds) 


MIL-STD —310F 
F Foot Drop 
10 rounds) 


MIL-STD —210F 
F Foot Drop 
C10 rounds) 


Fire 


Fire 
CG rounds) 


CG rounds) 


MIL-STD —310F 
Salt Fog 
G rounds) 


MIL-STD —331 
9 Foot Drop 


MIL-STD —331 
9 Foot Drop 


G rounds 
unpackaged 


CG rounds 
unpackaged 


Fire Fire 


CG rounds) CG rounds) 


Ve&RI = visual and radiographic inspection 


TEST CHALLENGES 


=e Schedule 
& Concurrent work 
B® Incorporating packaging changes into work flow 
ms Large Number of Tests 
BE 44 IM tests total 
B 105 Environmental tests total 
=e Volume of Data 
mg Hazard Classification of Mixed Smoke Pallet 


mw Packaging Problems At High Temperature 


® Non-hazardous off gassing produced overpressure 
® One way check valve installed 


mg Support Dunnage Problems During Vibration 


® Aft support broke down during transportation vibration at 
temperature 


® Dunnage redesigned and rounds repackaged 


mw Tests Repeated 


TEST RESULTS 


mu Planned Tests Completed, Regression Testing Underway 
m Packaging changes 
& IM signature, ammunition performed well 


@ Gun firing at NSWCDD and Hawthorne, NV 


m Leverage Test Facilities and Capabilities 
B Schedule 
B Idea sharing 


m Quick Response of PMO and Prime Contractor 
Invaluable 


= Communication = Flexibility 


Questions? 


OPERATIONAL ANALYSIS OF 
CBM+ TO ENHANCE SYSTEM 
SUITABILITY 


SCENARIO:PROBLEM 
SETTING 


ASSETS:VEHICLES, A/C; for MISSIONS 


BECOME FAILURE-PRONE, “UNHEALTHY” WITH WEAR, 
USAGE, “AGE” (RANDOMLY) 


ASSUME: HEALTH CONDITION “KNOWABLE” (SOMETIMES.... 
WITH ERROR ) 
—MONITOR HEALTH CONDITION (e.g. helo rotor vibration t) 


DIAGNOST. SYMPTOMS (DS): COST | 
AVAILABILITY | 

(PROVIDED CBM+ SYSTEM FUNCTIONS & WITH FEW 
MISTAKES!) 


Client: e.g. Army Aviation& Missile Cd., Redstone Ars.; 
Analyst. Data Whse. 


Condition Indicator (Cl) 
Development Summary 


Demonstrated Capabilities: 


Vibration Cls Determined From Analysis of Frequency & Energy Data Recorded By 
Embedded Sensors 


Cl Development Is Iterative And Requires High Quality Field Data To Determine 
Normal and Abnormal Behavior 


Cl Development Is Further Enhanced With Bench Test Data 
Each Cl Is Tailored For Individual Fault Modes 
The Complexity Of Cl Development Varies With: 

- The Number Of Fault Modes 

- Fault Occurrence Frequency 

- The Monitoring Capability For That Mode 


Benefits To The Warfighter: 


Teardown Inspections Are Used To Confirm Cl Thresholds 
Replace Manual Inspections With Active Monitoring 


Objectives: 


Extends Time Between Overhauls & Extends Service Life 
Increases Safety 


—W 
MISSION STARTS 


OUTCOMES 
WITHOUT/WITH CBM+ 


>> 


DEGRADED; 
UNDETECTED 


DEGRADED; 
DETECTED 


—— + 


FALSE POSITIVE; 
(NO ACTUAL 
DEGRADATION) 


Se Sbs pao seo > 


FAILURE (UNDETECTED) 


SUITAB. SUCCESS) 


MISSION ENDS; NO FAIL 

(MISSION CAPABLE: 

NO CBM+ (OR 
CBM+ 
MISSION ENDS/ABORTS; | INOPERATIVE) 
LONG, HIGH-COST 

REPAIR (MISSION 


INCAPABLE) 


MISSION ENDS/ABORTS; 
SHORT, 

LOW-COST REPAIR 
(MISSION INCAPABLE) 


MISSION ENDS/ABORTS; 
SHORT REPAIR (MISSION 
INCAPABLE) 


CBM+ (ERROR, 
MISTAKE-PRONE) 


MISSION ABORTS 


FALSE NEGATIVE 


Additional CBM+ Failure Modes 


¢ CBM+ Physical System Failures 
Prognostic Errors 
— CBM+ False Positives (No Actual Fault) 


— CBM+ False Negatives (Actual Undetected 
Fault) 


¢ T&E of System, Including CBM+ = IVHM, 


VITAL! 
— “End to End” 


Periodic Overhaul vs. Prognostics 
(IVHM or CBM+) 
Previous Work 


¢ [IDA: FCS] (Macheret, Koehn, & Sparrow) 
— CBM+ system is perfect but not all (Series) 


system components monitored 
— CBM+: 


KNOWN (!) Time from Prognostic— Failure 
(NO ERROR) 


— Result: CBM+ Cost], AVAIL. t vs. Periodic 
Overhaul (/f sufficiently many sub-systems 
successfully monitored) 


Periodic Overhaul vs. Prognostics 
(IVHM or CBM+) 
Previous Work 


¢ [BOEING] (Z. Williams, S. Cooper, 
J. Vian) 


— Integrated Vehicle Health Management 
(IVHM)=CBM+ 


— Fault isolation time log-Gauss; outliers. 
Simulation 


— Result: Optimistic assumptions — 
Availability t, Cost | 


Preventive Maintenance 
(Including CBM+) 


¢ Text: Gertzbach, |. Reliability Theory with 
Application to Preventive Maintenance 
(Chap. 4, Sec 4.2) Springer 


Present Model 


CBM+ subsystem imperfectly reliable: subject to 
functional/ “physical” failure and repair 

lf CBM+ subsystem up prior to a mission & produces a 
signal, the system will undergo repair/replacement 

— True positive: System would have failed during mission 

— False positive: System would not have failed during mission 

lf CBM+ subsystem up prior to a mission & does not 
produce a signal, the system is used on the mission 


— False negative: System fails during the mission (catastrophic 
failure) 


— True positive: System completes mission 
Independent, identically distributed missions 


Availability 
Parameters 


¢ Mean repair times (Multiple of Mission 
Times, e.g. 4 hrs.) 
— CBM+ subsystem failure: 10 (40 hrs.) 
— Detected failure: 15 (60 hrs.) 
— False Positive: 5 (20 hrs.) 
— Catastrophic Failure: 35 (140 hrs.) 
Mean number of missions between 


operational system failures (not CBM+) 
— 100 (400 hrs.) 


0.4 


Operational Availability 


Operational Availability 


Oo 10 20 30 40 50 60 70 


Mean # of missions to CBM+ subsystem failure 
Measure of CBM+ Reliability 


80 


—@ CBM+ false 
positive rate=0 


—=™- CBM+ false 
positive 
rate=0.05 


—— CBM+ false 
positive rate=0. 1 


—-— CBM+ false 
positive 
rate=0.15 


system Operational Availability 
Depends Upon 


Reliability of the CBM+ subsystem 

The false positive rate 

The rate of true positives (repair time of 
failures) 

The rate of false negatives (repair time is 
larger for catastrophic failures) 


Number of Systems 


¢ Systems operate and are repaired 
independently of each other, and have the 
same parameters. 


¢ There is a need for 250 systems to be 
available that can be assigned to missions 


Number of Systems Required 


All mean repair times 
twice as large 


Number of systems 


Number of Systems Needed for the Expected Number Number of Systems Needed for the Expected Number 
of Systems Up Equal to 250 of Systems Up Equal to 250 
Mean Repair Times Twice as Large 
-= CBM+ false n -= CBM+ false 
positive positive 
rate=0.15 ' rate=0.15 
1] 
i. E, 
| |-—CBM+ false 4 —— CBM+ false 
A positive rate=0. 1 ie positive rate=0. 1 
a: 3 
| rie 
Q 
| |->CBM+ false = + CBM+ false 
positive rate=0 2 positive rate=0 


Mean # of missions tween CBM+ subsystem failures Mean # of missions tween CBM+ subsystem failures 
(CBM+ Reliability) (CBM+ Reliability) 


14 


Number of Systems Needed 
Depends Upon 


Reliability of the physical system 
Reliability of CBM+ subsystem 
Mean repair times 

Rate of false positives 


Suitability 


¢ Not just a “requirement”: Essential for 
Mission Success 


¢ Affordability issue: 
Reliability of CBM+ & System (A/C) 7 
Spare (Logistics) Cost | 


Fixed Budget B 


¢ Develop/Test Physical System & CBM+ 
subsystem 
-MTTF 
— Operational Availability 

¢ Use remaining budget to buy systems to 
be fielded 


¢ Mean number of fielded systems up 


Tradeoffs 


¢ Less $ spent on development/testing 
— Buy more systems 
— Less operational availability 
¢ More $ spent on development/testing 
— Buy fewer systems 
— More operational availability 
¢ Fraction of development/testing budget 
spent on CBM+ 
— Less: operational availability | 


Decision Variables 


¢ Amount of budget to spend on 
development/testing 


¢ Fraction of development budget 
— Physical system 
— CBM+ subsystem 


¢ Rest of budget to buy systems 


Expected Number of Fielded Systems Up 


700 


600 
500 


400 


== 
1 2 ae 
OO 


Expected number up 


Fraction development budget spent to develop physical system 


Z4U 


Remarks 


¢ The maximum expected number of fielded 


systems available to start a mission is 
obtained by allocating (about) 40% of the 
total budget to development/testing 

The best allocation of the development 
budget: 

— 30% system development 

— /0% CBM+ subsystem development 


Conclusions 


¢ CBM+ has the promise to improve system 


reliability and to decrease maintenance costs. 


However: 


¢ CBM+ can introduce additional failure modes. 


¢ Reliability of CBM+ & System f 
Spare (Logistics) Cost | 

¢ Developmental and operational testing of the 
system MUST include the CBM+ subsystem. 
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Larry Graviss, Jacobs Eng. 
Committee Chairman 
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DOT&E, 


Co-Chairman 


James F. O’Bryon 
Chairman, NDIA T&E Division 


13 March 2007 
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NATIONAL DEFENSE ISOC TRIAL AS OCTATION 


STRENGTH THROUGH INDUSTRY {TECHNOLOGY 


BEF! TB a6 
pa OS ICOTE Organizing 
Principles 


The objectives of the ICOTE are to: 

«Provide a forum for discussion and exchange 
of T&E views from Defense and the 
Industrial Base in a non-attribution 
environment 

«Gain feedback from senior industry 
representatives and OSD leadership 

«Discuss OSD and service policies which 
affect relationships with industrial suppliers 

«Discuss emerging issues / policies in 
government and industry which affect the 
readiness and capabilities of U.S. defense 
system producers 

«Cooperate on various projects of mutual 
benefit to the ICOTE participants 


ICOTE 


Current Membership 


CURRENT NDIA T&E 
ICOTE PARTICIPANTS 


NDIA PARTICIPANTS 
Lt Gen Larry Farrell, USAF (Ret) Pres & CEO, NDIA 
Maj Gen Barry Bates, USA (Ret) VP Operations 
Sam Campagna; Director of Operations 
James O’Bryon: Chair T&E Division 


GOVERNMENT MEMBERS 
Dr. Charles McQueary DOT&E 
David Duma DOT&E 
Dr. Ernie Seglie DOT&E 
Michele Williams NGA 
Jim Streilein ATEC 
Brian Simmons AEC 
Mike Crisp DOT&E 
Maj Gen James Myles ATEC/CG 
Maj Gen Robin Scott AFOTEC CC 
RDML Bill McCarthy OPTEVFOR 
Colonel Mike Bohn USMC 
Colonel Debra Dexter DISA 
Dr. Steve Hutchison DISA 
Steve Whitehead OPTEVFOR 
Jerry Kitchen AFOTEC 


| 


NATIONAL DEFENSE INDUSTRIAL ASSOCIATION i 7 O | E 
STRENGTH THROUGH INDUSTRY & TECHNOLOGY 


Current Membership 


CURRENT NDIA T&E 
ICOTE PARTICIPANTS 


INDUSTRY MEMBERS 
Oscar Arroyo Raytheon 
Ray Lytle Raytheon 
Brian See ATK 
Regis Luther Armor Holdings 
Steve Zink Oshkosh 
Jim Ruma General Dynamics Land Sys 
Martin Peryea Bell Hellicopter 
Parker Horner EWA 
Bill Keegan WBB 
Gary Bridgewater SAIC 
Bill Shane Boeing 
Jim Vosper Boeing 
Doug Pearson Lockheed Martin 
Joe Sweeney Lockheed Martin 
Jon Neasham Cubic 
Gene Fraser Northrop Grumman 
Tom Quinn BAH 
Larry Graviss Jacobs Engineering 


NATIONAL DEFENSE INDUSTRIAL ASSOCIATION 
STRENGTH THROUGH INDUSTRY & TECHNOLOGY 


- MEETINGS 


The ICOTE will meet at the call of the 
Chairman and Co Chairman at sites and 
times convenient to the members. 


Minutes of the ICOTE will be provided to 
all members in the form of Agendas and 
action items as determined by the Director 
of OT&E. 


Topics of interest for consideration by the 
ICOTE will be solicited from industry and 
government members. 


BEES 0 
tweens ICOTE Themes for 
The past year 


Themes for 2006 /2007 
Net Ready KPP 
Testing in a Joint Environment 
Long Term Tactical Armor Strategy 
Testing in an Evolutionary Acquisition 
Environment 
M&s in OT&E 
Reliability Sustainability 
Sustainability KPP 
Contractor Logisitics Support 
Information Assurance 
ATD Supportability 
Sustainability and Evolutionary 
Acquisition 
Joint Rapid Acquisition 
Reports 
¢ DAPA Report 
¢ Net Ready KPP Study Report Mar 2006 


Headquarters U. S. Air Force 


Integrity - Service - Excellence 


23 Annual National 
Test & Evaluation 
Conference 

Mr. Dave Hamilton 


Service T&E Executives Speak 
15 Mar 07 


U.S. AIR FORCE 


«39 Outline 


U.S. AIR FORCE 


= AF T&E Update 
a AFSO21 Initiatives 
= Early Involvement 


= Suitability 


3/16/2007 Integrity - Service - Excellence 


\4 Air Force Missiles Fired from 
MA UTTR vs WSMR 


U.S. AIR FORCE 


a Utah Test and Training Range (UTTR) 
= AF Owned and Controlled 
m Ease Scheduling Conflicts 


m™ Reduced Cost — Fewer Program Schedule Slips 


3/16/2007 Integrity - Service - Excellence 


\4 Weapon System Evaluation 
hil Program (WSEP) 


U.S. AIR FORCE 


= Annual Brief to CSAF 


m Directed Increased Utilization of Guns Across CAF 

= 83 FWS to Investigate Options to Combat Banner 
m Significant Increase in Missile Firings 

= Improved Confidence Levels 


= Combat Archer Requested ~150/year 
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\ Z 
W Aerial Targets 


U.S. AIR FORCE 


= AFSAT (BQM-167A) 
= Replaces two legacy subscales (BQM-34 & MQM-107) 
w Flight test has been delayed due to reliability issues 

a OF-4 


= Resolved Control Amplifier Issue Which Caused a 10 Aug 06 
Crash At Holloman AFB 


= Resumed to NULLO (Not Under Live pul Operator) Ops 
g Inventory 4 


a Air Superiority Target 


m@ AOA In Progress - ECD Mar 07 
= Program funded for FY08 start 


—— a | 
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\4 EDO Advance Capability Pod 
iA (ACaP) Jammer 


U.S. AIR FORCE 


Purchased 6 ACaP Jammers, Support Equipment and 
Spares in FY05 for $11.65M From EDO Corporation 


= Improved Electronic Attack Threat Representation for 
Aggressors to Better Support CAF Training Requirements 

m 1550+ Total Aircrew Exposed to DRFM EA, 500+ Fighter 
Crews 

= “Best EA Training Ever Experienced” Red Flag Aircrew 
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KY 
YW AFSO21 - 5 T&E Initiatives 


U.S. AIR FORCE 


= Early Tester Involvement 
m Enhance Value of Test to Acquisition 
m Enhance T&E Efficiency 


a Integrated DT & OT 
m Reduce # of Stop Tests and Failures in OT&E 
m= Deliver Warfighter Capability Sooner 
a Information and Data Management 
m Enhance T&E Workforce Effectiveness and Development 
= Reduce Late Defect Discovery 
= Eliminate Cost Overruns 
a Optimizing Test Data Requirements 


= Minimize Data Collection & Duplication 
a Use Common Data Repository 


3/16/2007 Integrity - Service - Excellence 


Jf Early Involvement and 


fieaiicout Integrated Testing Work 


Y 


Small Diameter Bomb 


= Integrated Test Applied to 
Initial Seamless 
Verification Test (DT/OT) 


= Enhanced Operational 
Relevance During CT and DT 


= Maximized Opportunity to 
Use Early Test Data for 
IOT&E 
= Results: Fewer Weapon 
Releases Required for 
IOT&E 
=» JDAM = 132 Weapons 
= SDB = 28 Weapons 
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\7 3 Major Processes Aligned -- 
KA Requirements, Acquisition, & T&E 


U.S. AIR FORCE 


“Collaboration between the requirements, acquisition, DT&E and OT&E 
communities is key.” SAF/AQ (Dr. Sambur) and AFOTEC/CC (MGen Peck), April 02 
MS MS MS 


A © & IOC FOC 


Concept Technology Operations 
Refinement Development & Support 
al, a 


AFI 10-601 | i 

(Requirements) mgr @igcc : 
Capabilities Based Requirem: eee velopment (AFI 10-601 : 

(Acquisition) | 

AFI 99-103 


(Test & Evaluation) 


Capabilities Based Test and Evaluation (AFI 99-103) 
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ets Suitability 


U.S. AIR FORCE 


=u Defense Acquisition Objective: 
m Field Operationally Suitable Systems 
m Available for Combat when Needed 
= Reliable Enough to Accomplish its Mission 
= Does not Impose an Undue Logistics Burden 
= Operational Test and Evaluation is Statutory 


=m Required to Assess the Effectiveness and Suitability of Defense 
Systems Under Consideration for Procurement 


= Suitability has a Large Impact on 
Effectiveness 
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\ ¥f 
YY Suitability Drivers 


U.S. AIR FORCE 


a Suitability Deficiencies have been 
Responsible for Many of the Fielding 
Problems with Newly Acquired Systems 

= Crucial Suitability Issues are not Adequately Identified Early 
= Not Addressed in Operational Test Plans 
mw RAM are Driving Suitability Deficiencies 


Suitability Should be Assessed From 
Early Development Through Fielding 


Sz 
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<9 Summary 


U.S. AIR FORCE 


a Field Operationally Suitable Systems 


= Enhanced early involvement and information 
sharing will allow us to identify problems early, 
engage the contractor to find fixes sooner, with 
the goal of ensuring systems are ready to enter 
dedicated OT&E with a high probability of 
passing.* 
= Collaboration is Key to Transformation 
m Requirements — Acquisition — Test communities 


Support Quality Acquisition 
Through Credible T&E 


1) 
AFOTEC Ownership of Test Compression Target Programs,” Joint SAF/AQ (Sambur) -- AF/TE (Manclark) memo, 8 April, 2002 
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Revisiting Quantitative Methods for 
Evaluating Training Programs for 
Systems Undergoing Operational Test 
(OT) 


Dr. Christopher D. Hekimian 
Sr. Systems Engineer and Policy Analyst 
AIC 


4501 Ford Ave., Suite 330 
Alexandria, VA 22302 

Tel: 703 499-0518 

E-mail: hekimianc@saic.com 


Ms. Laura Chan 

Sr. Engineer/Analyst 

SAIC 

4501 Ford Ave., Suite 330 
Alexandria, VA 22302 

Tel: 703 296-0057 

E-mail: laura.w.chan@saic.com 


About the Presentation... 


We shall revisit a method that can be used to quantitatively 
assess the efficacy of training programs 


The purpose is to provide a basis to state, withina 
prescribed degree of confidence, whether a change in 
fielded system performance can be attributed to technical 
issues or to training 


A classical, hypothesis-based approach, where operator 
“expert” or “trainee” -status is the independent variable, is 
described 
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The Problem 


A system seems to perform well during 
Developmental Testing (DT) but when made 
operational, there is a notable decline in 
performance... Is the problem with the system 
or with the training? 


Common Methods for Assessing Training 


e Surveys, Questionnaires 
e Interviews 


¢ Focus Groups 


These provide qualitative assessments that are 
not only based upon the respondents’ 
experiences during OT, but also upon the 


aggregated experience, knowledge, feelings and 
attitudes of the respondents. 


Drawbacks of Common Methods for Assessing Training 


Subjectivity 
— What one person’s assessment of what is good/bad or acceptable/unacceptable is 
likely to vary based upon numerous things, including the background of the 
individual responding to the question 


¢ Compounding the problem is that survey question responses ranges are seldom 
“anchored” to something objective 


Internal Validity 
— Due to ambiguity or diverse definitions, interviewer or survey questions may not be 
measuring what the evaluator is interested in 
¢ Scientifically valid surveys are pre-tested and aligned in order to ensure validity. 
External Validity 
— The results of the sample survey may not be extendable to the larger population of 
all potential users of the system 
Bias 
— Some respondents may have a conscious or subconscious bias towards a given 
question response 
Apathy 
— Some respondents may resent the additional demand on their time of one more 
surveys. They may not use care in responding to the survey questions 


Advantages of Common Methods 


Requires little or no additional testing 
because responses are based upon results 
obtained while collecting other measures 


Provides the opportunity for other relevant 
insights regarding the system or related 
DOTMLPF to be collected 


DOTMLPF: Doctrine, Organizations, Training, Materiel, Leadership 
And Education, Personnel, And Facilities 


Experiment Design Method for Determining the Efficacy of 
Training Programs 


Hypothesis testing allows one to make an authoritative statement like : 
‘the training did not have a significant impact on the OT performance results 


The ability gained through training is assigned as the independent variable 


— This is done using two test methods: 


e Realistic Scenario Testing (RST) performed by experts under conditions that replicate 
Operational conditions to the greatest degree possible - - establishing a performance 
baseline 


¢ Operational Testing (OT) performed by actual end users in operational conditions or ina 
realistic operational exercise 


Experiment involves the test results of two groups, the experts performing 
RST and the end users performing OT 

Are the performance results of the experts during RST that much better than 
the real operators during OT? 

Validity of the results depends on the faithfulness with which the RST 
replicates the OT 


Defining the Performance Metric 


The data elements that would normally be used to assess 
effectiveness of a system are used to assess training efficacy 
All variable types can be used 
¢ Nominal 

e¢ Not associated with a value, just a label or categorization 
¢ Ordinal 


e Associated with a range of integer values such as with a Likert scale (i.e., 
1 to 5 scale) 


e Ratio 
e Value can be any real number 
Performance metrics that are returned in averaged amounts need to 
be partitioned 


e j.e., instead of considering one test of 100 trials, ten averages of ten 
different trials are computed 
e The need to do this will soon be made plain 


Governing Assumptions 


Systems used during OT and RST are essentially the 
same 


Experts are sufficiently experienced with the system 


RST environment and test conditions are sufficiently 
similar to OT condition 


Test subjects (if any) are similar between RST and OT 


Effects of operational stress during OT are negligible or 
somehow duplicated in RST 


Statistical Significance 


Randomness effects all experimental results. Simply looking at the overall 
number of successes and failures in a test is an insufficient basis for 
conclusions. One must determine if the test results show statistical 
significance. 


Significance tests account for the possibility of a test result occurring due to 
chance alone. Various tests for significance have been developed and the 
correct test must be applied based upon the type of variables and the data 
distribution. 


Two of the most common tests for significance are the t-test, which is used for 
parametric (normally distributed ) data; and the Chi-square test, which is used 
for nominal (categorical) data. 


If a significance test indicates that there is less than a 5% chance that the 
observed difference in results could be due only to chance, then usually a 
significant effect of the independent variable upon the results is noted. 


Significance Calculations 


The statistical tests of Chi-Square (for nominal variables) and t-test (for ordinal, ratio, or 
averaged data) are available to test the following hypothesis: 
— “Training did not have a significant impact on the performance of the system during 
OT” 
The t-test will provide a measure of the likelihood that the performance averages taken 
from two test groups were taken from distributions with the same mean 


OT Results Combined RST and OT Results 


RST Results 


Given the dispersion of 
the data for.each of the 
two groups, how likely is 
it that the total data was 
drawn from two separate 
distributions (A) or from 
a single distribution (B)? 


Statistical Significance- Chi-Square [1] 


Applicable to nominal test result data 


Microsoft Excel and other spreadsheet programs can 
accommodate Chi-square calculations 


“Help files” provide usage and application instructions 


|B Book1 
By 
———————— 
 Gimy ee ore ee 
a Able to assemble system <5 minutes 
PSS OBS (Unable to assemble system <5 minutes | 
| 4 [RSTExpected Values __[OTExpectedValues [Result 
| 5 [(A2+B2)(A2+A3)/(A2+B2+A3+B3) _|(A2+B2)(B2+B3)/(A2+B2+A3+B3) |Able to assemble system <5 minutes _| 
| 6 [(A3+B3)(A2+A3)/(A2+B2+A3+B3) _[(A3+B3)(B2+B3)/(A2+B2+A3+B3) [Unable to assemble system <5 minutes | 
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Insert Function (2) 


Search for a function: 


Type a brief description of what you want to do and then 
dick Go 


Or select a category: | Statistical v 
Select a function: 


AVERAGEA 
BETADIST 
BETAINV 
BINOMDIST 
CHIDIST 
CHIINV 

v 
CHITEST(actual_range,expected_range) 

Returns the test for independence: the value from the chi-squared 
distribution for the statistic and the appropriate degrees of freedom. 
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Statistical Significance- Chi-Square [2] 


1. Populate a table as shown. Note that 
columns A and B of rows 2 and 3 are the 
results of tests. The same columns in rows 5 
and 6 are derived as shown from the test 
results above them 


2. Use INSERT => FUNCTION and then select 
the STATISTICAL category. Select CHITEST 


3. The wizard will allow you to select the cells of 
the test results (4 cells, A2,A3,B2,B3 in our 
example), and then the expected values that 
you have inserted, each, respectively 


Clicking OK will return the result of the Chi- 
Square Test 


5. The function returns the probability that the 
difference in performance between the RST 
and OT tests could have occurred due to 
chance as opposed to due to training effects 
(i.e. CHITEST value of 0.05 indicates a 5% 
probability that the difference in test results 
was not due to training issues +) 


1.) Based upon all governing assumptions 


2.) Typically, 5% is the threshold (CHITEST >.05) where 
one would assume that the effects of training were not a 
significant factor in the difference in performance 


Statistical Significance- t-Test [1] 


Applicable to normally distributed data such as those 
returned from survey results or laboratory 
measurements 


Microsoft Excel and other spreadsheet programs can 
also compute t-test results 


= Book1 

ee DS: ee aa: 
i Kill Rates using new Targeting System 

2 [Trial |RST Results by Experts JOT Results by End Users 


COON DNF WKH = 


_ 
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>ring “1” for the test type a paired t-test could be performed 


d t-test could be used in order to compare the ordered 
of two data sets for one group, one before and one after 


vould be a direct, single group test on the efficacy of a training 


sults could be of interest but would need to be interpreted 
due to the possibility that the results of user “habituation” 
system is the causal agent and not training (internal validity) 


Statistical Significance- t-Test [2] 


1. Populate a table as shown. Note that the columns B and C 
might be populated with test results for individual test 
trials, averaged trial results for a particular test subject, or 
averaged survey results 


2. Use INSERT => FUNCTION and then select the 
STATISTICAL category. Select TTEST 


3. The wizard will allow you to select the cells of the RST test 
results (B3:B7 in our example), and then the OT results 
(C3:C17). The number of entries in each column need not 
be the same for our example 


4. Enter a“2” for the number of tails to use for the test. This 
will account for the possibility that the end users over or 
under perform the experts 


5. Enter “3” for the test type as there is no reason to assume 
that the variances of the two data samples will be the 
same 


6. The function returns the probability that the difference in 
performance between the RST and OT tests could have 
occurred due to chance as opposed to due to training 
effects (i.e. TTEST value of 0.05 indicates a 5% probability 
that the difference in test results was not due to training 
issues *). If TTEST >0.05 assume training was not a 
performance issue 


* Based upon all governing assumptions 


Partitioning Technique for Averaged Performance 
Parameters (1 of 3) 


Biometric System Performance Example 


Administrators of a fingerprint access control device are concerned 
as to whether the high False Accept Rate (FAR) of their system is 
due to poor system performance or might be a training related issue 


FAR is an aggregated performance parameter that is compiled from 
a large number of trials... In this case, fingerprint authentication 
transactions 


The methods described previously are not suitable for aggregated 
performance parameters because the data are not nominal, and the 
FAR is already an averaged value and thus, there would be no 
multiple trials to average and analyze. 


Partitioning Technique for Averaged Performance 
2 of 3 


In order to employ the method, the two FAR tests (RST and OT) of 
500 test subjects each are partitioned into twenty FAR tests (ten 
each RST and OT) of 50 test subjects each. 


Every effort is made to ensure that the systems, the . 
environment, and the test subject demographics are similar 
between RST and OT 

Partitioning is employed in this case so the ten RST and OT FAR 
tests can be analyzed using a t-test 

Partitioning can be by order, as in this example, or randomized 
The number of test subjects and trials between the RST and OT 
tests need not be the same 

FAR tests by their nature typically require a large number of test 
subjects 

Overall system FAR calculations should be based upon all 500 
trials for each test- - with no attempt being made to combine the 
individual FAR results into one representative one 


Partitioning Technique for Averaged Performance 
3 of 3 


RST outperformed OT. The t-test indicated that there was an 28% 

likelihood that the RST and OT FAR test results could have been 

drawn from distributions with identical means 

— Because of uncontrollable factors such as stress during OT, and 
the relatively large likelihood that the performance difference was 
due to chance (28%), Analysts do not attribute the lapse in 
performance to training issues. 


— Technical solutions are emphasized over training ones 


The same data may be analyzed in an identical manner for different 
aggregated parameters (such as False Reject Rate) 
—~ The results should not considered to be confirmatory because the 
two analyses are not based upon independent data 
— If the <5% threshold for significance is met, then training problems 
would be indicated 


vantages and Drawbacks of Quantitative 
Methods for Evaluating Trainin 


Advantages: 

e The methods are objective in terms of data collection, in 
terms of analysis and in terms of interpretation of results 

e The methods are associated with specific levels of 
confidence for conclusions 


Drawbacks: 
¢ Cost of conducting RST in addition to OT 


Conclusions 


Survey methods can be supplemented, confirmed, or refuted using quantitative 
methods 


Method for attributing lapses of performance to either system or training was 
discussed 
— Experimental design using RST and OT establishes training as the 
independent variable 
— Nominal performance metric (Chi-square) 
— Ordinal and ratio metrics (t-test) 
— Aggregated performance metric (t-test) 


Other applications for using quantitative methods for analyzing training programs 
include: 

— Side by side comparison of different training approaches 

— Pre and Post training analyses 

— Analysis of effectiveness of modifications to training programs 


Thank you! 
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Expanding the Realm of Possibility 


Introduction 


= Reliability-based methods are those that 


e Use the probability of failure as a criterion in the design 
process 


= These methods contribute to suitability, effectiveness and 
sustainability by 
e Improving system performance 
e Increasing operational readiness 
e Reducing unnecessary intervention or maintenance 
e Managing spare parts inventories 
e Reducing technical and operational risk 
m Other benefits of these methods 
e Provide predicted performance across a range of metrics 


e Support decision-makers by highlighting trade-offs in 
performance and RAM 


4p) Expanding the Realm of Possibility 


The “Magic” behind the Methods 


= These methods involve 
e Applying probability distributions to uncertainties 


e Using physics-based modeling to assess the impact 
of these uncertain factors on system performance 


e Balancing system design features and inspection 
intervals against risk 
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Physics-based Probabilistic Analysis 


PRObabilistic Function Evaluation System 


Compute: 
- Risk 


ALN 


- Reliabilit 
Geometry y 


- Reliability sensitivities 
- Response probability 
Se distributions 
- Failure samples for 
maintenance simulation 


Environment 


Material e4 


Other Uncertainties 


Uncertainty analysis based on fast |" 
probability integration and 
advanced random simulations 


Most Likely Failure Point 


Optimize: 

- Risk and Reliability 
- Maintenance Plan 

- Test Plan 


apg @ Failure 
va 4 Sampling 


Model Uncertainties Develop High Fidelity basieche 
and Variability Computational Models Apply State-of-the-Art probabilistic methods 
Wy 
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Reliability-based Methods that support 
Suitability, Effectiveness and Sustainability 


= Reliability-based multidisciplinary optimization 


(RBMDO) 
= Reliability-based damage tolerance (RBDT) 


4p) Expanding the Realm of Possibility 


[ee 
Reliability-based Multi-disciplinary 
Optimization (RBMDO) 


= Optimizes performance subject to multiple reliability- 
based constraints 


= Incorporates multi-disciplinary objectives/models (e.g., 
payload, aerodynamics, shape parameters, weight...) 


= Accomplishes higher performance over independent 
optimization of each discipline 


4p) Expanding the Realm of Possibility 


RBMDO Application — Aircraft Wing Design 


m 3804 finite element nodes 


3770 finite elements (2222 
shells + 1548 beams) 


22 material properties 
47 shell element properties 


= NASTRAN model of 
Advanced Composite 
Technology (ACT) wing 


=m Baseline aircraft: proposed 
190-passenger, two-class, 
transport aircraft 

=m Critical Design conditions 
derived from DC-10-10 and 
MD-90-30 
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Rear spar panels (R3) 
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Another Application — Radiation Detector Sustainment 
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Reliability-Based Damage Tolerance (RBDT) Framework 


Fully Integrated Finite Element stress, Fracture Mechanics life and ProFES analyses 


Principal Structural 


Element Model 


- stress model 


- life model 


Usage 
Load Spectra 


Flaw or FOD Stress 


icles. 


Material Modeling Error 


Inspection time 


Fracture Mechanics Damage Tolerance 


Update distribution 
Crack Size] after maintenance 


Initial 
Flaw 4 


Time (Flight hours) 


ProFES 


Most Likely Failure Point 
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Non-Destruc. Inspection Planning 


Critical Size 


Cum. Probability 
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Crack Size 
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Expanding the Realm of Possibility 


Reliability-Based Damage Tolerance (RBDT) Methodology 
for Rotorcraft Structures 


= Project sponsored by FAA 


= Critical structures must maintain very 
small probability of failure ee Pacis 
= Supplement current “safe-life” design 


approach (which tends to be too 


conservative) Inspection Life/No. of Optimized schedules for 
= Systematically treat _ Schedule inspections max. risk reduction 
variability/ uncertainty In: Safety Measure Safety margin Reliability 
e usage, load, flaw, material, Other Variables | Bounds or Safety | Distributions 
geometry, modeling error, Paci 
defect detection capability 


= Maintenance planning for: 


e Non-Destructive Inspection, 
inspection frequencies, 
repair/replacement 


= Has wide applicability to structures 
with material or manufacturing flaws 


e Select appropriate NDI interval 
e Optimize sustainment strategies 


4 ARA 


— Without Inspection 
— With Inspection } Max. risk reduction 


Prob. of Fracture 


Flight Hours 
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RBDT Application -- Rotorcraft Spindle Lug Model 


Reference R = 0.25m, Thickness = 67 mm, Initial flaw size = 0.4 mm 
Random variables for the lug model 


Distribution Mean SD Cov (%) 
Thickness, ¢ (mm) LN 28 0.14 0.50 
Max. load (N) LN 145000 10000 6.9 
Initial flaw size (mm) User-defined 0.074 0.0224 30.2 
Delta Ky, LN 48 4 8.33 
Life scatter LN l 0.1 10.0 
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RBDT Yields Optimal Inspection Schedule 
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[ee 
Reliability-based Damage Tolerance Application 
Pipeline Maintenance Optimization 


= Corrosive environments cause metal losses 
= Two major failure modes with uncertainties 


e Burst can cause a high consequence but is less 
likely to occur 


e Leak has a low consequence but is more likely to 
occur 


g In-line inspection devices can detect significant defects 
= Options to maintain integrity at different risk/cost: 


e Defect monitoring using high resolution ILI 
devices (cost issue) 


e Repair/replace sections (cost issue) 


e Reduce operating pressure (production loss) In Line Inspection 

e Corrosion mitigation (cost issue) Magnetic Flux Leakage and other ILI 

ee devices can travel through pipelines to 
= Objective detect metal losses 


e Develop maintenance optimization software using 
ILI] data and probabilistic failure and cost models 
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Pipeline Maintenance Example 


Prob. Defect 
Model 


Cost 
Model 


Maintenance 
Options 


Opt. Maintenance 
¢ Repair/No Repair 

¢ Repair/Replace 

¢ Repair Methods 

¢ Future Inspection 
Schedules 


Prob. 
Corrosion 
Model 


Risk/Reli-Based |. 
Decision Model 


Prob. 
Burst/Leak 
Models 


Risk is minimized if 
the next inspection 
and repair time is 3 
years from the last 
0.6} inspection 


Pf at 20 Yrs 
[o) 
ol 


Pipeline Maintenance 


Optimization Methodology 


Optimal Inspection Time at 3 Yrs 
l [ 
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Another Application — Robotic/Unmanned Systems 
Design 
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Implications for T&E 


LONGITUDINAL STRESS (GPa) 
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Wrap-up 


= Accomplishments 


e Tools for taking a reliability-based approach to design and 
sustainment that can result in cost savings and risk reduction 


e Traceable predictions of reliability and performance changes 
e Computationally fast and efficient methods 
=» Challenges 
e Scalability of approach 
e Understanding material failure properties 
e Cascade of variable and interrelationships 
> Non-unitized structures 
e “User friendliness” 
> Enable use by decision-makers 
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ieee } 
Path Ahead 


= Mature/prove methods for specifying probability 
distributions 


e ARA’s Klein Associates Division (cognitive 
scientists) 


= Refine the understanding of the impact and interaction 
of the uncontrollable random variables 


= Accelerate collection of data to inform physics-based 
models 


= Continue to evolve cost- and time-effective testing 
approaches for verifying reliability 
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NAVSEA Hazard Assessment Testing of the SM-3 Block IA Missile 


DAHLGREN 


Outline 


¢ Description of SM-3 Block IA missile 
e Hazard Assessment Test Program 

e Test methodologies 

e Summary of results 

e Lessons-learned 
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SM-3 Block IA Missile 


e Sea-based component of the Ballistic Missile Defense system 
e Launched from Vertical Launching System of DDG-class ships 
— Approximately 22 ft length x 13.5 in diameter 
e MK 72 booster ~21 in diameter 
— Contains ~2065 lbm propellant 
— Designed for MK 21 MOD 2 VLS canister 
¢ Total mass of all-up round ~6300 Ibm (i.e., missile and canister) 


¢ Exo-atmospheric intercept using Kinetic Warhead = 


a 


Guidance 
Section 


_ 


MK 72 Booster 


Steering Control Section 


MK 104 Dual Thrust 
Rocket Motor 


Nosecone 
Staging 
Third Stage ssembly 


<x 
Rocket Motor 


Kinetic Warhead 
(inside Nosecone) 
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Hazard Assessment Test Program for SM-3 


1999 een )S 220086 CX. 2005/2006 
SM-3 Block 0 SM-3 Block | SM-3 Block IA 


Environmental Tests 
(encanistered missile) 


28-Day Temperature 
& Humidity 


28-Day Temperature 
& Humidity 


28-Day Temperature 
& Humidity 


Transportation Vibration 


Transportation Vibration : : : 
Shipboard Vibration 


Shipboard Vibration 
Launch Shock 


Transportation Vibration 
Shipboard Vibration 


4-Day Temperature 
& Humidity 


4-Day Temperature 
4-Day Temperature & Humidity 


& Humidity 


Near Miss Shock 


40-ft Drop 
40-ft Drop Horizontal 
Aft-end down Functional Tests 


of Components 


Insensitive Munitions Tests 
(individual sections) 


Bullet Impact 


Fast Cook-off Slow Cook-off Third Stage Rocket Motor 
Third Stage Rocket Motor Third Stage Rocket Motor Kinetic Warhead 
Kinetic Warhead Kinetic Warhead 


Fragment Impact 
Third Stage Rocket Motor 
Kinetic Warhead 


Fast Cook-off 
Third Stage Rocket Motor 
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28-Day | 4-Day Temperature and Humidity (T&H) Test Method 


Encanistered missile cycled between hot/humid and cold environments 
— Conditions based on environmental profile for logistics life-cycle 
e¢ +130F with 95% RH for hot/humid environment 
e -20F for cold environment 
— 1cycle includes 24-hr (min) exposure to each environment 
Tests conducted using programmable environmental chamber 
Test methods identical except for duration 
— 14 cycles for 28-day T&H; 2 cycles for 4-day T&H 


Nominal RH 
(uncontrolled) 


Temperature (°F) 


5 40 45 50 


Relative Humidity (%) 
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Transportation Vibration Test Method 
e Encanistered missile subjected to random vibration IAW MIL-STD-810 
— Simulate transportation by truck over improved roads 
— Input applied through 3 orthogonal axes; 1 axis at a time 
— 3 hr/axis duration to simulate 3000 miles over-the-road transport 


e 2 hydraulic actuators used to provide input at each PHS&T skid 


J See Ea 


Longitudinal Axis 


Transverse Axis 
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Transportation Vibration Input Profile 
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Shipboard Vibration Test Method 
e Encanistered missile subjected to random vibration to simulate shipboard 
environment 
— Input profile and duration based on shipboard measurements in VLS cells 
e 4-200 Hz frequency range 
e Input spectrum encompasses full range of ship speeds and sea states 
e 39-hr/axis to simulate anticipated deployment durations 
— Input applied through 3 orthogonal axes; 1 axis ata time 
e System-specific tailored test requiring approval from NAVSEA O5T 
— Most systems tested IAW MIL-STD-167 
e Sinusoidal vibration across 5 — 50 Hz frequency range 
e Accomplished using UD-4000 electrodynamic shaker 
— Special fixtures used to provide input at correct interfaces with canister 
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Shipboard Vibration Input Spectrum 
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Shipboard Vibration Test Setup (Transverse Axis 


Deck 
Transverse Shipboard S. nterface 
Vibration Fixture = Fixture 
All-Up Missile in_ ! 
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Shaker 
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Lateral Support 
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Near Miss Shock Test Method 


e Accomplished using DS-3 Shock Machine 
— Large-displacement, pendulum-type impact shock machine 
— Highly-tunable design 
¢ Continuously adjustable pendulum impact velocity 
e Unique adjustable fixture permits input through 3 principal axes of item 
e System-specific tailored test requiring approval from NAVSEA O5P3 
— Alternative to Heavyweight Test (i.e., “Barge Test”) of MIL-S-901 
— Input levels tuned to actual field measurements 
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Near Miss Shock Test Setup 


Deck Interface 
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Tuning of Input for Near Miss Shock Test 
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Pendulum drop height — peak velocity 
Programmer pad thickness — initial acceleration 

Brakes & Springs — initial pulse duration; magnitude of neg. velocity 
Obliquity/clocking — Longitudinal and Transverse components 


NAVSEA Hazard Assessment Testing of the SM-3 Block IA Missile 


DAHLGREN 


Representative Response Data 
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Configuration of Kinetic Warhead 


Ejector Assembly 


MTA Thrusters 


Sustain Grain 


Cryogenic Gas Bottle 
(Nitrogen @ 6000 psi) 
Pulse 1 Grain 


Guidance Unit 


Pulse 2 Grain 


¢ 3 propellant grains in Solid Divert and Attitude Control System (SDACS) 
— Pulse | grain is TP-H-3510 propellant 
— Pulse 2 grain is TP-H-3511 propellant 
— Sustain grain is TP-H-3512 propellant 

¢ SDACS case is graphite-epoxy composite 
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Configuration of Kinetic Warhead for Insensitive Munitions Tests 


Ejector 


Nosecone Guidance Unit 


| : fs << End Cap 
=. § (simulated Guidance Section) 


Cryogenic Gas Bottle 


SDACS 


X-SMDC 
e Kinetic Warhead assembled to simulated Guidance Section shroud 


— 13.625-in OD annular aluminum cylinder with 42-in thick aluminum closure 
plate 
— Guidance Unit simulated using high-fidelity mass model 
— Cryogenic gas bottle present and fully-charged 
e Block IA Nosecone installed over Kinetic Warhead 


— Secured to simulated Guidance Section shroud in same manner as 
tactical missile 


— All Nosecone explosive components present 
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General Configuration of Third Stage Rocket Motor 


Ignition Tube Paes lay 


Pulse 2 Grain 


Toroidal Cold 
Gas Bottle 


Composite Case 


¢ 2 propellant grains 

— Pulse | grain is TP-H-3518A propellant 

— Pulse 2 grain is TP-H-3518B propellant 
¢ Case sidewall is filament-wound graphite-epoxy composite 
¢ Toroidal cold gas bottle contains pressurized nitrogen 
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Configuration of Third Stage Rocket Motor for Insensitive Munitions Tests 


Closure Plate 
page Dwg SKSG6B-LP-21 2) 


Fwd End Cap 
_ (Raytheon Dwg SKSG6B-LP-213) 


e 


Third Stage Rocket Motor 


_ Aft End Cap 
Forward — (Raytheon Dwg SKSG6B-LP-21 4) 


Closure Plate 
(Raytheon Dwg SKSG6B-LP-21 2) 


¢ TSRM assembled with end caps to simulate adjoining missile sections 


— Each end cap is 13.72-in OD annular aluminum cylinder with %-in thick 
closure plate 


— Aft end cap secured using 4 explosive bolts 
e Replicate configuration of tactical missile 


nV TZN Hazard Assessment Testing of the SM-3 Block IA Missile 


DAHLGREN 


Bullet Impact Test Method 
e Item impacted by three (max) 0.50-cal AP projectiles 
— Velocity of 2800 + 200 ft/s 
— Bullets fired at 50 ms intervals using three 50-cal Mann barrels 
e Trajectory of bullets perpendicular to longitudinal axis of test item 
— Bullets aimed to pass through center of SDACS propellant in KW 
— Bullets aimed to pass through Pulse II grain and ignition tube in TSRM 
e Instrumentation and data collection IAW MIL-STD-2105B 
— Gun firing times 
— Bullet velocities 
— Air shock 
— High-speed video record of events 
Post-test recovery and characterization of remains 
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Fragment Impact Test Method 

e |tem impacted by three %2-in mild-steel cubes 

— Velocity of 6000 + 200 ft/s 

— Cubes launched using 60mm smoothbore gun and unique FRP sabot 
e Trajectory of cubes perpendicular to longitudinal axis of test item 

— Aimed to pass through center of SDACS propellant in KW 

— Aimed to pass through Pulse II grain and ignition tube in TSRM 
e Instrumentation and data collection IAW MIL-STD-2105B 

— Cube velocities 

— Air shock 

— High-speed video record of events 

— Post-test recovery and characterization of remains 
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Fragment Impact Test Setup 
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Fast Cook-Off Test Method 


e Item suspended above pool of burning JP-5 aviation fuel 
— Average flame temperature >1600F 
— 30-ft x 30-ft fuel basin used to ensure complete immersion within flame 
e Instrumentation and data collection IAW MIL-STD-2105B 
— Flame temperature 
— Air shock 
— Video record of events 
— Post-test recovery and characterization of remains 
e Pretest modeling to predict time to reaction 
— 1-D model to examine radial heat transfer through case sidewall 
— Examined two bounding flame temperature conditions 
¢ 1600°F average flame temperature 
¢ 2000°F average flame temperature 
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Fast Cook-Off Test Setup 
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Predicted Temperature at Liner/Propellant Interface During TSRM Fast Cook-Off 
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Summary of Test Results 


Test 


28-Day Temperature & Humidity | No safety-related anomalies 
Transportation Vibration No safety-related anomalies 
Shipboard Vibration No safety-related anomalies 
4-Day Temperature & Humidity No safety-related anomalies 
Near Miss Shock No safety-related anomalies 


Bullet Impact Kinetic Warhead: Type IV reaction (deflagration) 
Third Stage Rocket Motor: Type Ill reaction (explosion) 


Fragment Impact Kinetic Warhead: Type IV reaction (deflagration) 
Third Stage Rocket Motor: Type Ill reaction (explosion) 


Fast Cook-Off Third Stage Rocket Motor: Type IV reaction (deflagration) 
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Lessons Learned 


Multi-shaker setup used for Transportation Vibration test introduces additional 
issues related to phase control 


— Currently not explicitly addressed in MIL-STD-810 
— Accepted / best practices still evolving 


Not possible to achieve same input levels at both ends of canister in Shipboard 
Vibration test due to fixture dynamics 


— Fact-of-life constraint for single-shaker setup using large, complex fixture 
— Problem most pronounced at higher frequencies 


— New state-of-the-art facility at NSWC/Dahlgren will enable multi-shaker 
testing in vertical orientation 


Near Miss Shock test demonstrated capability to replicate real-world triaxial 
shock input to large encanistered missile using pendulum-type shock machine 


— Potential alternative to “Barge Test” for some systems 
¢ Subject to approval by NAVSEA O5P5 on case-by-case basis 
e May reduce system design risks 
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More Info 


e Test program documented in two NSWCDD technical reports 


— NSWCDD/TR-06/47, Standard Missile - 3 Block IA Hazard Assessment 
Test Results 


¢ Draft currently in final review 
e Expect publication and release within a few weeks 


— NSWCDD/TR-06/48, Standard Missile-3 Block IA Near Miss Shock 
Qualification Test Report 


Estimating Durations and Trials to 


Success in Test Programs 
¢ Click to edit Master text styles 
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Traditional T&E Schedule Estimation 


USA] T&E Processes 


a 
aise | 28K 


| « T&E programs are inherently risky: 
— Individual WBS elements carry considerable schedule risk 


— There are complex relationships between test objectives, 
outcomes, and future work 


— Each outcome has complex risks and consequences 
Not intuitive; difficult to scope 


Traditional T&E Schedule Estimation 


¢ T&E schedules are estimated in variety of ways 


— Depends on time, data, precision needed, guidance from 
program office 
¢ Traditionally, 3 ways to estimate T&E schedules: 
— Factors based upon historical data from analogous systems 
— Parametric Schedule Estimating Relationships (SERs) 


e Linking some characteristic system parameter to historical 
schedules 


— Detailed bottom-up estimates 


« Predominantly driven by projected staffing 
requirements 


¢ Usually assumes only planned tests 


Traditional T&E Schedule Estimation 


« Accounting for risk and unknowns is a historical challenge 


for T&E cost estimation 


¢ Use of historical analogies often fail to adequately account 
for important distinctions in the new system 


¢ History-based parametric analysis can reasonably estimate 
the T&E schedules, but provide no information about 
critical test elements 


¢ Bottom-up estimates provide a wealth of detail on 
individual test elements 


— Doesn’t account for additional unplanned tests resulting from 
test failure 


— Schedules are consistently inaccurate and always low 


¢ Generalized Activity Network (GAN) analysis supports the 
development of more accurate bottom-up SERs 
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¢ A Generalized Activity Network (GAN) is: 
— Acyclical directed process modeling diagram (an 


extension of PERT) 
— The modeling capabilities of GANs include: 
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GAN Junctions 


GAN Receivers GAN Transmitters 
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Monitor 
Supplier 


Build In-house 


or outsource? 
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Sim 


ulate < 
..that can compute completion time and 


cost for complex spending programs. 


Our research shows that these simulations provide 
a surprising amount of insight, even with few inputs - 


« Modeling process: 


— Build a network diagram (GAN) to describe possible program 
execution paths 


— Estimate parameters: establish random distribution(s) 


— Require probabilities for feedback loops or other event 
outcomes 


— Create a discrete-event simulation for that network 


¢ Parameter estimation: 
— Task durations, cost, and risk levels can be based on: 
¢ Build-up estimates, calibration with historical data, subject matter 
expertise 
¢ Often apply a Weibull distribution (Gladstone-Miller 2002 
DODCAS) to deterministic estimate 
— Feedback probabilities can be calibrated with historical data 
from similar programs or subject matter experts 
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Example: Repeat-Until-Pass Test GAN 


LSA] Repeat-Until-Pass Test GAN 


Recover from moderate failure tmod 
Recover from major failure C maj 
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Fail & Prod 
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Ctest 


Uprep 


tweaks 
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Most likely task durations in blue 
Event outcome probabilities in red 
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Example: Repeat-Until-Pass Test GAN 


LA ¢ Durations for Preparation and Testing: 


— Uniform Random Variables 
— Expectation 1 day & Rangelday:  U(0.5,1.5) 


¢ Durations for Recovery from Test Failure: 


— Minor Failure: U(0.5,1.5) Exp. Value: 1 day 
— Moderate Failure: U(1.25,2.75) Exp. Value: 2 days 
— Major Failure: U(2.0, 4.0) Exp. Value: 3 days 


— Note: Dispersion also increases with failure severity 
¢ Duration for activities following success Is 0 
© Psuccess = Praiture = +9 
° P= .8; mod wal oe 

— 10% of all failures are moderate, and 10% of all failures are major 
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¢« Performed Monte Carlo Simulation 
¢ Expected Duration for Test Success: 2.8 days 


¢ Large right-tail dispersion due to geometric distribution from 
inclusion of a probability of test failure 
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TMI GAN Advantages 


¢ Hierarchical: can describe and analyze system at any level of 


detail 


¢ Flexible: supports evaluation and decision-making at all levels 
¢ Model iterative processes 


¢ Can provide more information than simple time/cost estimates 
— Complete distribution; eliminates need for separate risk analysis 
— Identify potential problem activities for risk mitigation 


¢ Often provide useful insight during both design (diagramming) 
and analysis (simulation, analytic equations) phases 


¢ Provides a single integrated approach for understanding task 
interdependencies; identifying high-risk activities; and 
incorporating funding constraints 
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GAN Disadvantages 


¢« “Uniqueness” problem 


— Data cannot be used for calibration if too program-specific 
— Breadth of data as important as depth of data 


¢ May suffer from subjectivity of expert opinion data 
— Problem of all bottom-up estimates 


¢ Requires detailed program data 
— Data necessary for calibration 
— Calibration necessary for meaningful schedule estimates 


¢ “Familiarity” problem: Although growing, GANSs currently 
not widely used for cost analysis 
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EMI) Calibrating GAN Probabilities 


¢ We consider two common GAN feedback 


DrOCeSSes 


— The One “P” Case 


e Single feedback loop with a constant probability of 
SUCCeSS 


e Preliminary results included in MORS presentation 


— The Two “P” Case 


e Successive attempts after the first failure possess a 
constant, but higher, probability of success that the first 
test trial 


e Presumes that most of the major problems are at least 
identified after recovery from initial failure implying a 
higher probability of success for subsequent trials 
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GAN Probabilities: One “P” Case 


Rework 


Fail 


Finalize 


Typically, probabilities of success or failure driven by expert 
opinion 


¢« Assumptions 


— Well defined, common test event for commodity/system 
— Access to historical data from similar systems 


Probabilities can be appropriately calibrated by historical data 
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GAN Probabilities: One “P” Case 


¢ Considering simple test-block GAN: 


— Trials occur until a success is achieved (with probability P for 
each trial) 

— Let X be the number of trials until the first success 

— xX is ageometric random variable with parameter P 

— Specifically, 


E[x]=— 


p 


¢« Assuming historical data (of sample size n) on number of trials 
from similar systems can solve for single p* that minimizes the 
sum of squared errors between the expected number of trials 
predicted by the GAN, E[X] ,and the historical data 
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LMI GAN Probabilities: One “P” Case 


1 
¢ Thus, if ar and {b,,b,,b3,..-,, } 


are the set of outcomes representing the number of trials for 
independent outcomes of the same GAN, we wish to: 


min y (x—b.)° 


i=1 


subject to: x =1 


¢ Conveniently, the global minimum is simply »», 
the mean of the historical data, yielding: p* =| —— 
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One “P” Case: Proof 


¢ Since our problem is only over one dimension, we can simply 


consider looking at the derivative of the function with respect to x 


n n 


Yi(x-b) = YX? =2bx+b/) = nx? —2x5°b, +b? 
i=l i=l 


i=l i=l 


¢ Taking the derivative of this expression and setting it to zero, 
we get that: 


2nx-2) 5=0 => x=—— 
i=1 n 


¢« Thus, we can estimate p* by simply by taking the inverse of the 
average of the outcomes of the trials 
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LMI GAN Probabilities: One “P” Case 


¢ This simple, straightforward result is powerful because 


analysts can easily objectively calibrate GAN probabilities 


¢ Further, in absence of historical data, analysts should seek 
— Unbiased expert opinion on “average” number of tests until 
SUCCeSS 
— Should produce better estimates of realistic probability of 
success than directly asking for them 
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¢ Probability of success on first test: P, 


¢ Probability of success on every other test, conditional on 
first test failing: P, 
— Might expect P,>P, due to knowledge of what failed, 
additional effort spent on that item, etc. 
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GAN Probabilities: Two “P” Case 


¢ Consider a test event with the following historical data: 


# Trials 1st Trial 2nd Trial? 
Historical until Success? (Did the 1st # of "P2" 
Program Success (Yes=1, No=0) _ trial fail?) Trials 


| el 
Ee 


2 
3 
4 
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6 
7 
8 
9 
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¢ We could calculate a single probability, op, using the previous 
technique 


— Method of calibrating P, and P, should reduce to One “P” case if 
probabilities are constant 
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GAN Probabilities: Two “P” Case 


- Let x, and x, be decision variables and {b,,b,,b,,...,b, } 


historical data. 


- Let b; =0 ifthe first trial failed and b, =1 if it succeeded and 
assume that there are J successes. 


¢ Let b; represent the number of subsequent trials with a 
probability, p, ,of success 


¢ As before, we wish to minimize the sum of squared errors 
between the expected number of trials predicted by the GAN 
and the historical data for each decision node: 


DX -b,)° +> (x, —b})° 
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GAN Probabilities: Two “P” Case 


¢ Wecan minimize each sum separately, yielding x, and x, , and 


thus our P, and P, 


¢ Using the data from our example we produce the probabilities: 


P=x,=0.3 
Pp, eS gaia 
X, a 


¢ Monte Carlo testing demonstrates method to provide robust 
estimation of data generating process even when P, = P, 
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Weibull Enveloping Distributions 


¢ Evaluate feasibility of estimating hyper-geometric processes 


through an enveloping Weibull distribution 


¢ Why? 
— Hyper-geometric processes and Weibull distributions are 
similarly right tailed 
— Weibull distributions are well understood throughout cost 
community and, thus, better accepted than GAN feedback 
simulations 


¢ Study Objective: See if its possible to fit a Weibull to a feedback 
process under ideal conditions, ie. when we actually possess 
full Knowledge of the process and relevant statistics 


35 


Weibull Enveloping Distributions 


« Method for matching GAN simulation results through Weibull 


distributions: 
— Create GAN simulation to generate hyper-geometric data 


¢ Can be calibrated with different underlying probabilities and duration 
distributions 


¢ Produce test data from 10,000 trials to ensure robust characterization of 
data generating process (can employ asymptotic theory) 


— Optimize Weibull parameters to fit the distribution to generated data 
e Identify appropriate metrics to define “goodness of fit” 
¢ Perform optimization upon selected objective function 

— Evaluate “best-fitting” Weibull predictions to simulated results 
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Weibull Enveloping Distributions 


« Population Data: Data Generating Process 


— Relatively straightforward parameters to maximize probability of 
successful fit 


— Trial Durations ~ Uniform (2,6) 
— Feedback Probabilities: 50/50, 80/80, 30/30, 50/80, etc. 


¢ Goodness of fit 

— Cost and Schedule Estimates typically reported at the mean 
[expected value], 50% CDF, and 80% CDF 

— If we know two of the three, we can optimize the selection of 
Weibull parameters (a,8) to minimize the error between the 
prediction for the third metric and simulated data 

— If we assume we only know the expected value of the data, we can 
optimize parameters such that we minimize the joint error between 
the 50% and 80% CDF 


a7 


LMI 


Characteristic Result 


GAN Simulation and Fitted Weibull 
0.4 12 
0.35 
OOOO OOS SH SOSH SH OOOOH OOOOH OOOH 1 
0.3 — 
: see? 
0.25 bs + 0.8 
> ¢ 
4g 925 | - 0.6 
2 
é 0.15 + 
| r 0.4 
0.1 + 
r 0.2 
0.05 - 0 
. + 0 
NST ODMDONTODAONTtTO AON + 
NmoFTonraeoocodoatnonrood 
dod da aA dtd aA oH ANN 
Months =] )PDF 
---@-- CDF 
—m— Fitted Weibull 


Weibull Enveloping Distributions 


¢ Our analysis on a variety of simple GAN simulations indicates 


that it is not feasible to adequately envelope a GAN feedback 
loop with a single Weibull distribution 


¢ Enveloping Weibull performs worse under more complex, 
realistic assumptions such as Normal or Weibull distributed 
durations in the population data generating process 


¢ For afew specific cases we were able to fit a Weibull witha 
relatively small mean squared error, e.g. a prediction error of 
less than 20% 
— However, calibration was made with our complete knowledge of 
the underlying data generating process, which we would not have 
with real data. 
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Weibull Enveloping Distributions 


¢ Successful fit only indicates that it is feasible for a Weibull to 


approximate a specific feedback process not that it can 
actually be implemented with statistical confidence 
— Wouldn't actually possess information on the expected value, 
50%, or 80% CDF with real data from which to optimize Weibull 
— Weibull parameters would be calibrated, as always, from outside 
of the estimated process 


— Fitting arbitrary points of a CDF does not necessarily indicate a 
minimization of the error between the Weibull and simulated data 
mass functions 


¢ We recommend the continued use of GAN feedback loops to 
simulate feedback processes, such as testing. 
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Pharmaceutical Testing 


Laboratory Testing 

Animal Testing: Effectiveness & Safety 
FDA review: Decision to allow clinical trials 
Clinical trials: Effectiveness & Safety 


Very costly consequences of unexpected 
serious side effects after distribution, 
patient use 


Development can be stopped at any time 
for cause (e.g. VIOXX) 


Continuum of Testing of Military 
Systems: 


Objective 
Assure Adequate & Timely T&E Funding 


Materiel Solution(s) to fill “Capability Gap” 
— Early Operational Assessments 
¢ Effectiveness & suitability 
¢ Operational and technical risks? 
Development 
— Frequent assessment of progress 
— Test in realistic environments as early as possible 
Effectiveness and Suitability of the entire system 
after fielding 


— Lessons learned 


Need Material Solution(s) for 
Capability Gap 


Early operational assessments of 

proposed solutions (M.&S. ) 

— Input to design effort 

— Factors important in design 

— Measurable cost-effective improvements to 
mission capability (Evolutionary Acquisition) 

— Anticipate need for Dynamic Action by Blue to 
Red Adaptation 
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Modeling and Simulation for 
End-to-End System Evaluation: 1 


Size of budget & its allocation 

— Research and Development (R&D) 

Prototypes; Developmental Testing, Subsystem Integration/DT 
— Developmental & Operational Test and Evaluation 

Production & Procurement Costs 

— CONOPS 

— Sustainability; logistics 

Early analysis of schedule implications, including design 
upgrade, technology risk, and testing 


— lf time requirement tight to field a block, the system more 
expensive (and slow) to field: problems found during OT&E, or 
worse, in the field (engineering design changes) 

¢ GEN Welch: “rush to failure” for THAAD 
¢ “If you don’t need it to work, | can ship it now” 


Modeling and Simulation for 
End-to-End System Evaluation: 2 


Adequate developmental and operational T&E funding 


Preview system capability in an operational 
environment: Model 
¢ Unexpected vulnerabilities in effectiveness and suitability 
¢ Plan test design 
— Seed failure modes to be detected 


Preview test design 
¢ Use experience with similar systems 
— Government needs access to contractor performance data 


¢ Establish a DoD database containing lessons learned during 
system development 


¢ Red Threat 
M&S Not a substitute for testing: guidance 


system Development and 
Demonstration 


¢ Demonstrate system performance in its 
intended environment using competing 
prototypes 

¢ T&E to assess technical progress for 
operational utility 

¢ Early operational assessments 


¢ ID technology risks 
¢ Provide operational user impacts 


DT results generally better than OT 
results 


¢ “The PM's rationale is that they will tell me that 
in past tests they have never had that problem 
before. Well, you never had that problem before 
because we, the soldier, use the equipment in 
the mud, and in the rain and we use it every day 
by the average soldier.” 


BG Honore 
1st CAV 


Bring Mature Systems to 
Operational Test 
¢ Systems fail because they are immature in 


design or manufacture, not (always!) 
because the fundamentals are bad 
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Operational Testing 


¢ EARLY (and Affordably Often) under 
representative real world conditions 


— Early discovery of problems permits fixes 
sooner & less costly; e.g. system weighs too 
much for (transport to) intended use in the 
field 


— Learn from training and exercise opportunities 


Current Model 


¢ Single-use system possibly containing design 
defects (DDs) 


— Each system used for one field mission 


¢ Tests may discover DDs which are then 
removed (Reliability Growth) 


— Each remaining DD survives a test independently of 
previous tests 

— When DD activates during a test, the DD is removed 
(intention! Not always successful) 

— Test conditions— Probability of DD test survival 


Fixed Budget 


Includes 
— Testing & Removing discovered DDs 
¢ reliability growth 
— Buying copies for fielding 
¢ Copies may have DDs remaining 
¢ Remaining DD activates during field mission— mission fails 
¢ Cost of modification 


Tradeoft: 


— Spend less on reliability growth— can buy more systems, but it 
is more likely remaining DDs will activate during a mission 
(mission failure): redesign of the fielded system 


— Spend more on reliability growth— can buy fewer systems but it 
is more likely a system will finish a mission without DDs 
activating 
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Design Defects (DDs) 
(Simplified Distinction!) 
- Two types of DDs: DD1 & DD2 


— DD2s more difficult to activate during a test 
than DD1s 


¢ When a DD is discovered during a test, it 
is removed 


Tests 


¢ Type 1 (Early) 
— Less expensive 


— Not as effective at activating DD2s; as effective at activating 
DD1s 


— Less expensive to remove activated DDs of both types 
— Can choose effectiveness of early tests 
¢ More expensive early tests are more effective at activating DD2s 
¢ Type 2 (Late) 
— Expensive 


— More effective at activating DD2s (as effective at activating 
DD1s) 


— More expensive to remove activated DDs of both types 


Learning from Testing 


Continue to do Early Tests until there are r1 
Early tests in a row during which 0 DDs 
activated (r1 successful tests in a row) 

— Not learning more 

Then Late Tests until there are r2 Late tests ina 
row during which 0 DDs are activated 

— Not learning more 

ba remaining budget to buy systems for the 
ield 

Choose r1 and r2 so as to maximize the mean 
number of fielded systems (missions) for which 
no remaining DDs activate 


Number of Successes in a Row that Maximizes the 
Expected Number of Field missions in which No 
DDs activate (maximum run length=5) 


Cost per 
Early Test 


smaller 


larger 


Prob. Best 
Surv. DD2 | Success 
in One Run 


Early Test Length: 


Early 
Tests 


0.99 


(less 
effective) 


0.85 


(more 
effective) 


Best Mean # 
Success Field Mis- 
Pin sions with 
Length: 0 DDs- 
Late Tests | 2ctvating 


(more 
expensive) 


Remarks 


¢ More effective Early Tests — 


— Longer best run of successes for early tests 
(despite additional expense) 


— More reliability growth early 
— Fewer Late (more expensive) Tests 


— More field missions completed without 
remaining DDs activating 


Example: 
Naval Special Warfare Rigid-Hull 
Inflatable Boat (NSW RIB) 


Close teamwork early testing: OPTEVFOR and combat 
users 

Competing Vendor Prototypes developed & tested until 
acceptance trials and source selection 

Early testing operationally realistic: OT&E completed in 
source selection phase before production contract 

All operational requirements met or exceeded during 
combined DT/OT 

Early reliability experience for NSW RIB much better 
than for craft it is replacing 


Continuum of Testing 


Ensure adequate funding for T&E 
Provide operational insights throughout 
the development process 

Realistic testing environments as soon as 
possible 

— Mature systems to operational test 

Result: Field an effective and suitable 
system as early as possible and with less 
cost 
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Innovate, Execute, Excel 


Testing of the GPS SAASM 
End-to-End Functionality 
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Operational Weapons Platforms 
Without the Availability of 
the Signal in Space 


{to 13 March 2007 
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Overview 


¢ Background SAASM 
— What is SAASM 
— Status of Implementation 
— Test Capability Shortfall 
— Proposed Solution 
¢ SAASM-ISER Test Set up 
— Concept 
— Components 
— Test Strategy 
¢ Demo on Army HIMARS and Navy P-3 
¢ SAASM-ISER Phase II Interfaces 
¢ Benefits to the User 
e Recap 
¢ Questions 


“Excellence Through Innovation” 
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¢ New generation GPS Security Architecture 


¢ Extended functions 

¢e Black (unclassified) Keys 
¢ Over the Air Rekey (OTAR) 
¢ Anti-tamper design 

¢ Direct Y enabler 
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SAASM GPS 


¢ What does the User Get ? 
— Same Accuracy Performance 
— More Secure Military Ops 
— Simplified handling 
— More Capability 
— Over the Air Rekey 


¢ Signal in Space (SIS) awaiting 
Master Control Segment upgrade 
— User and Space Segment ready 


“Excellence Through Innovation” 
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¢ Cards developed 

¢ Integrated into box 

¢ Integrated into systems 

¢ Systems going operational 
¢ SIS not available yet 


fF - = “See 
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Testing Gap 


¢ GPS SAASM cards are tested at the chip 
level, receivers are tested at the box level 


¢ However, fully integrated system level testing 
was not being accomplished 


¢ So, there is a Testing Gap of full functionality 
at the integrated system level 
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Systems Engineering from 
Mission Need to Operational Part 


Testing Gap 


Specifications 


System Integration 
(HV or GPS) 


Low-level Test 


Development 


Integrate 


Development Phases 
Subsystem (UE or GNC) 


UNCLASSIFIED 
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Lessons Learned 


eee eee Fe 
Mae” 10a “a 


¢ Most acquisition programs begin test phase 
with high level of confidence and optimism 
in the expected outcome, but... 


¢ In practice most every test program 
experiences unanticipated ‘glitches’ that 
require correction 
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ae 


INTEGRATED 
SYSTEM 
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Problem 
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¢ Platforms that use SAASM-based receivers 
cannot be tested easily at the system level 


— On-orbit signals are not yet available 
— Hard for operational platform to come to lab 
— Anechoic chamber testing is very costly 
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Wait for SIS ? 
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¢ Even with SIS a gap in test capability exists 
— Can’t expect full functionality testing from SIS 
— Difficult to control scenarios during tests 
— Repeatable scenarios probably not available 


— Anomaly resolutions; lack the ability to duplicate 
specific conditions and signals 


« Needed: 


¢ Capability to test before & after going operational 

¢ Ability to simulate and control all the SIS scenarios 
¢ Mobility to travel to the test platform 

¢ Connectivity - easy interface with various platforms 
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¢ SAASM Integrated System Evaluator and 
Reporter - (SAASM-ISER) 


— Configure a precision GPS signal simulator to 
generate and exercise the extended functions on 
the SAASM-equipped platform 


— Make it a mobile test capability that can travel to 
the user’s location; palletize and van equipped 


¢ Little to no down time on operational asset 


— Allow user’s platform systems to be run unaltered 
in their operational configuration — antenna hood 
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&% Phase | SAASM-ISER Components 


¢ RF Hood (FRPA), GPS Simulator with PC NavTEL, 
Test Van 
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ANALYSIS 


fo 


REPORT 
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acd Quality Control 
Signal Monitor 
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Test Strategy and Approach S 
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¢ SAASM scenarios and data collection 
requirements are coordinated with customer 
— Determine best scenarios, sequence, and data 
collection sources for SAASM-ISER tests 
¢ Scenarios are a subset of SAASM test vectors 
— Synchronize scenarios to platform test location 
¢ Consistent with INS geo-position and time 
— Test the hood for RF leakage prior to tests 
¢ Add shielding if necessary 
— Collect preplanned Data to include: 
¢ GPS State, Code Track, PVT, FOM, C/No 
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HIMARS Initial Demonstration 
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¢ Demonstrated first SAASM-ISER proof of concept 
and basic capabilities on the Army’s High Mobility 
Artillery Rocket System (HIMARS) in 2005 


¢ Desired test vectors successfully accomplished on 
a fully integrated and operational weapon system 


HIMARS GMLRS 
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Antenna Hood 


GPS Simulated signal 
scenarios are fed to test 
platform’s GPS antenna and 
adjacent antenna tied to 
quality control monitor system 
to ensure signal integrity 


Hood Mounted 
over 2 GPS 
Antennas 
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SAASM-ISER Test Van 
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| Fully Equipped: 

- Elec Power Systems 

- Rack Tie-Downs 

- Dual Air Conditioning 
- Insulated 


- Climate Controlled 


Laptop 
Monitors, 


DAGR 


Signal quality i 
and baseline 
monitor 


Simulator 
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Navy P-3 Checkout 


Performed SAASM and GPS RAIM tests on 
operational P-3 using SAASM-ISER 
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> Phase Il SAASM-ISER Concept S 
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RS-422 


"“—_—e = * 


Mil Std 1553 


Pylon Data 
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SAASM-ISER Block Diagram S 


¢ RF feed into platform antenna(s) 
¢ RS-422, Mil-Std-1553, CDU data monitoring 


DAGR , 
M 


Integrated 
2K Rcvr Status Navigation 
signal 


monitor 
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SAASM-ISER 
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— Assessment methods: 


¢ Real-time assessment via cockpit displays 
¢ Data collected from receiver RS 422 
Instrumentation Port 


¢ Data collected from platform Mil-Std 1553 
bus 


— Verifies integrated navigation system 
functionality of unaltered platform and 
weapon interface 
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¢ Overall Benefits to the Warfighter are: 


— Reduces risk by substantiating technical 
performance prior to verifying system readiness 


— Resolves integrated system deficiencies early 
— Provides cost effective, short duration testing 


— Mobile - Can be performed on an operational 
platform at home station 


— Leverages 746 TS expertise in SAASM lab testing 
— Provides a means of anomaly resolution if needed 
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Recap 
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SAASM- Integrated System Evaluator & Reporter 


— Fills a gap in the government’s ability to perform 
end-to-end tests integrated SAASM systems 


— Provides for functional checkout of GPS SAASM 
extended functions on fully integrated weapons 
platform at low cost on home station 


— Provides means to duplicate/investigate anomalies 
SAASM-ISER Implemented on: 


— Army HIMARS 
— Navy P-3 
Follow on aircraft are in planning phase 
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Questions? 
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Backup Slides 
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Test Strategy 

HIMARS 3 Tier System 

Two FRPAs in Test 

Baseline Monitor 

SAASM-ISER Spirent, PC NavTEL 
Hood Detail 

Interface Cards 
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Test Strategy and Approach S 
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¢ SAASM scenarios and data collection 
requirements are coordinated with customer 
— Determine best scenarios, sequence, and data 
collection sources for SAASM-ISER tests 
¢ Scenarios are a subset of SAASM test vectors 
— Synchronize scenarios to platform test location 
¢ Consistent with INS geo-position and time 
— Test the hood for RF leakage prior to tests 
¢ Add shielding if necessary 
— Collect preplanned Data to be collected include: 
¢ GPS State, Code Tracking, PVT, FOM, C/No 


“Excellence Through Innovation” 27 if 


- CONSISTS OF THREE SYSTEMS INTEGRATED 


- Fire control system 


- Position / Navigation system (GPS / INS) 
- Launcher Weapon system (GPS / INS) 
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Baseline monitor antenna 
mounted next to platform’s 
antenna. Feeds monitor 
SAASM receiver (DAGR) 


Baseline SAASM monitor 
ensures the quality of the 
signal and the scenario 
being used for the unit 
under test. 
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¢ GPS Satellite Simulator 


— Spirent STR 4760 
— Multi Channel GPS L1/L2, CA, P(Y) code 
— Generates GPS simulations 


— Certified by GPS JPO according to the 
Enhanced Validation Test Plan 


e PC NavTEL 


— Provides real-time bus control, monitoring, 
communications and display of simulation 


— Exercises various models and test scenarios 


— Communicates with Spirent over a General 
Purpose Interface Bus (GPIB) interface 
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FRPA RF Hoods 


1) 
2) 


Metal Enclosure & RF Absorbent interior 


Includes internally mounted FRPA radiator to 
provide simulated RF SAASM GPS signals 


Signal is received by UUT antenna and separate 
monitor FRPA under the antenna hood; 


System-Under-Test antenna 
Baseline FRPA connected to DAGR 


¢ Provides baseline monitoring and quality 
control for signal integrity of SAASM-ISER 
scenarios 
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SAASM-ISER Interfaces 


¢ Interface Cards: 
— Mil-Std-1553 
¢ Common SAASM Interface 
Change Notice 
— RS-422 
¢ IP for ICD-GPS-150 type messages 
— Weapon Interface 
¢ Mil-Std 1760 connector 
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SAASM-ISER (Phase II) 
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¢ Software module communications development 
— RS-422 (Instrumentation Port data) 
— Mil-Std-1553 
¢ Common SAASM ICN (IFC-SJICWG-001) Provides 
“common” SAASM interface data to Smart Mil-Std- 
1760 Weapons 
¢ Test and Validate 
— Conduct phase Il tests on proxy system 
¢ MAGR2000-GS or SAASM-EGI in lab testing 
— Conduct analysis 


¢ Work with platforms on special requirements 
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Testing the Latest 
Embedded GPSI/INS 
Hybrid Navigation System 
for the 
F-16 Fighting Falcon 


13 March 2007 


Jim Killian 
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Overview 


— 7.) mick 


F-16 LN-260 EGI Test Item 

Approach and Test Methodology 

Test Capabilities Available at 46 Test Group 
LN-260 Overall Test Objectives 

Sequence Selected for Test Beds & Assets 
Truth Reference System to be used 
Summary / Lessons Learned / Conclusions 
Questions 
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F-16 Fighting Falcon 
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Integrated INS & GPS 
Navigation System 
for USAF and European 
Participating AF (EPAF) 


AIRFORCE.COM 


— Attitude 

— Navigation (PVT) 
¢ Position 
¢ Velocity 
¢ Time 
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e Separate 
GPS and 
INS units 


INS: Ring Laser Gyro GPS: 5 Channel PPS 
Inertial Navigator Satellite Receiver 


Blended GPS/INS 


PVT Solution 
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e LN-260 NG Aircraft Navigation and Attitude 
— Inertial Navigation System (INS) <.8 nmi/hr RER 
¢ New Fiber Optic Gyros 
— GPS: RC GEM-VI, 24 channel L1 & L2 receiver 
— Single Unit; smaller size, weight, cost, power use 
¢ Embedded GPS in INS (EGI), with Kalman Filter 


¢ Provides 3 separate solutions: 
— INS only, GPS only, Blended 
¢ ‘Tightly coupled’ 


e Same Performance ? 


e Effective and Suitable ? 
— Requires T&E 
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Notional Test Approach 
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SENSORS: 


Accelerometers ~ Gs 
Gyros ~ Rotational Rates 


(GPS Receiver ~ Noise/Signal ) 
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Test Phases 
Test Bed Costs 
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Test Approach 


Crawl - Walk - Run 


Discover and correct issues early at a lower cost 
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Mature the system design 
¢ Methodically verify proper functionality of 
navigation unit 
— Physical and functional checkouts 
— Precision three axis table attitude and rotations 
e Establish baseline performance 
— Benign, controlled, repeatable environment 
— Recheck as appropriate 
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« Performance and Characterization 


— Gradually increase stresses on sensors and 
system to the specification limits while 
measuring performance in realistic profiles that 
are controlled and repeatable 


— The greater the dynamics and signal stress 
the more exposed existing problems become 


— Use state-of-the-art Truth Reference System 
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Test Location: 46 Test Group 
Holloman AFB NM 


Flight 
Test 


¢ State-of-the-art Test Capabilities 
¢ 746 TS; Central Inertial & GPS Guidance Test Facility (CIGTF) 
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¢ 5 Test Squadrons within 46" Test Group 
— 586 FLTS: Flight Test Squadron 
— 746 TS: INS & GPS Guidance Test Squadron (CIGTF) 
— 781 TS: National RCS Test Facility (NRTF) 
— 846 TS: Holloman High Speed Test Track 
— Det 1: White Sands Missile Range Test Agent 


Ment, 


¢ 746 TS/CIGTF: Has complete 
range of GPS & INS Navigation 
and Guidance test capabilities 
— Satellite Ref Station (SRS) 


— Mobile SRS (MRS) 
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Helicopter C-12 T-38 F-16, F15 
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Inertial and GPS Lab Facilities 


120” Precision Centrifuge 
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Test Objectives for LN-260 EGI S 
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Verify the F-16 LN-260 EGI performance complies with the 
published specifications and characterize significant 
aspects without specification 

Functional 

— Check physical, power and data properties 

— Demonstrate operation of interfaces and EGI modes 

— Baseline navigation outputs 

Performance 

— Evaluate the navigation performance under realistic operational 

and environmental conditions 

Characterization 

— Characterize calibration errors 
Areas of concern include effect of temperature, G and 
vibration on navigation performance 
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Test Bed Progression 
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Bench top physical and functional checks Low Cost 
Precision 3 Axis table position/rotations Preparatory 
— 24 position tests at temperature variations 


— Inertial sensor calibrations 


; ; ; Med Cost 
Modeling and Simulation; NavTEL Comprehensive 


Van land navigation vehicle, low dynamics = Contolled 


environment 
C-12 cargo aircraft, low-med dynamics 
Precision 3 Axis inertial calibration recheck 


Most realistic 


T-38 Fighter aircraft, med-high dynamics environments 


High Cost 


Environmental: altitude, vibration, high G increased risk 
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Precision 3 Axis: Position 
Rotation, Temperature Tests 


- Payload (200 Ib) 

- Accuracy (1 arcsec) 

- Gimbal Rates (+750 deg/sec) 
- Chamber (-55 to +85 deg C) 


- Functional Alignment and Navigation 
- INS Calibration Validation 

-- 24 Positions (1g Environment ) 

-- Rotation Rates ( 20 deg/sec ) 
- Calibration / sensor baselines 
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FL el al lh 


-Controlled Signal Injection 
ae Platform Model ——Sencor Model - INS & GPS profiles 
Satellite Gj <- fs - Models: Baro, Doppler, + 
Simulator ;, 4} Data Bus : . ' 
7 e . NavTEL [ae - Peele Navigation Ops 
ee = - Jamming (broad scope) 
| | WWW> i «ane - RAIM Integrity Monitoring 
m2 GPS Receiver - SAASM Security Functions 


EGI 


Navigation T&E Laboratory (NavTEL 
- Hardware-in-the-Loop Design 
- Trajectories (Real & Simulated) 
- Models: Sensor & Aiding, w/ Realistic Errors 
- GPS Simulators (Spirent 4760 / 7700) 
- EGI Simulator (CAST) Hybrid, > Fidelity 
- Select Parameter Controlled On the Fly 
- Interference Signal Generators (Jammers) 
- Wave Front Simulator (Multi-Element Antenna Test) 
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¢ Null Steering Antenna Tests 
¢e Embedded Jammer Approach 5 
e Jamming; up to 16 Sources a 
e Precise and Repeatable Tests ors gd 


¢ Coherent Arrival Vectors, GPS and Jamming 
¢ Controlled Signal Location, Timing & Phase 


3 Jammer example: 30, 60, 270 Controlled Reception Pattern Antenna 


System]: 1 ( -186,0000, -169,2725) 
J@ 3060 270 
0 


runt 
run2 
rung 


( } 
Coherent Wave Front Sim for Predicted Model Measured Results : 


GPS and Jamming signals ; 
“Excellence Through Innovation” 17 


- Operational - Live Performance S 


ee ee re 
Mar” 0a “a 


Low Dynamic Test Medium Dynamic Test High Dynamic Test 


Land Navigation Vehicle Cargo Aircraft (C-12J) Hi Performance (T-38) 
- Velocity (26.8 m/sec) - Velocity (140-250 kts) —_- Velocity ( Mach 1.1) 


- Acceleration (19) - Acceleration (2.59) - Acceleration (7.29 ) 

- Navigation Modes - Navigation Modes - Air to Grnd scenarios 
- GPS Jamming - GPS Jamming - Air to Air scenarios 

- Precision Reference - Precision Reference - Precision Reference 
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Field Jamming Assets 


ee ee 
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¢ GPS Interference; Performance under signal stress 


Jamming Vans 


WSMR Test Range 


Albuquerque 


High Gain Antenna 
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GPS Signal Jamming Stress S 


ee eee Fe 
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NavTEL Van, C-12 
Simulated Live Signal and 
dynamic or dynamic profile 

Static profile 
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- Environmental Lab Capabilities S 


ee eee Fe 
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¢ Test specification requirements not reached on previous test beds 


Altitude / Temp Chamber 


—_— 


Vibration test beds 


- Frequency/G controlled 
- F-16 vibration profile 

- M-60 Gatling gun 

- Up to 1750 Ibs force 

- 2-2000 Hz (50#, 70g) 

- 1 inch travel 


- Payload Navigating 

- Pressure Altitude 80K’ + 

- 50-60K’ planned 

- Temp to match altitude 
-100 deg to +350 deg 

- Humidity 5% — 95% 


“Excellence Through Innovation” 


Centrifuge 120” Radius 


- Payload (100 Ib) 
- Accuracy (1 ppm) 
- Acceleration (0.5 to 50 g) 
- 13.5 g planned 
- Test Item Fixture: 
- Fixed or 
- Counter- Rotating 


, ¥ 


ee eee Fe 
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(Approximate) 


400 


200 


Dynamics (g) 
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(C-12J/ Van ) 


Fighter Inertial Navigation 


System (FINS) 
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CRS - Subsystem Configuration Accuracy 


| Subsystem ] RMS Position (m) 
a 


[1][2] GPS Code 


[1][2] DGPS Code! 1.25<>1.75 | 1.0<>1.75 1.5<>2.5 


[4] RRS/STARS 1.40 / 0.14 1.00 / 0.10 1.7 | 0.17 
STS Absolute / Relative 0.071 / 0.0014 0.05 / 0.0010 0.087 / 0.0017 


SRS Range Constraints: 1300-500nm 750-100nm Differential GPS 


[ Subsystem ] RMS Velocity (m/s) 


mde Ae ae a 


[1] INS (EGI) 0.010 0.010 0.010 0.017 
ins (ESNu) | 000s | oaos | o00s | oie 


Attitude Accuracy: 20 arcsec (Roll, Pitch, Heading ) 
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SUMMARY 


¢ CRAWL 
— Benchtop physical and functional tests 
— 53Y 24 precision position, attitude and rotations 
— Baseline performance for inertial portion 

e WALK 


— NavTEL Modeling & Simulation 

— Van 2D low dynamics, low rates 

— Baseline GPS, INS and Hybrid solutions <= 
e RUN 

— C-12, 3D med dynamics, med rates 

— T-38, 3D high dynamics, high rates 
¢ Environmental 

— Centrifuge; Altitude chamber; Vibration profiles 

— Higher risk and stress modes = 


“Excellence Through Innovation” 


Lessons Learned 
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¢ Often PM tendency is to streamline the T&E 
phases due to Cost and Schedule pressures 


¢ Testing WILL find unanticipated problems, 
guaranteed. 


¢ A thorough and systematic government 
independent T&E approach will actually 
reduce ultimate cost and schedule by 
finding/correcting problems early 


— “Rely on Independent Government Test” 
— “Focus on Performance” 
(Words of Gen Randolph, JNC 2004) 
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CONCLUSION 


Fl ne lh 
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Follow well planned graduated test approach that 
manages risk and finds and fixes problems early 


Avoid cutting plan to save time or $, which often 
increases Cost and Schedule. 


Thorough Benchtop and 3 Axis table tests are 
valuable in uncovering problems early, prior to van 
and flight testing 


Match tests to realistic environment as practicable 
Apply increased stress in a controlled fashion 


Place high risk tests at the end to reduce possible 
early schedule impacts due to system failure 
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e LN-260 EGI 
¢ Wide range of test 
capabilities in house 
¢ Benefits of selected order 
— “Crawl, Walk, Run” 
— Facilitates testers familiarity with test item 
— Establish Baseline performances 
— Comprehensive; identify & isolate problems early 
— Risk managed to reduce ‘re-fly’ schedule impact 
— Mature system design in most cost efficient method 
— Continuity with same team of experts throughout 
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Questions? 
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Back Up Slides 
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¢ CIGTF Reference System Block Diagram 
¢ CRPA Wave Front System Block Diagram 
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vA... at Satellites mn. =~ 


Reference Pallet Configuration ¢ CRSP y- 


a 


Reference 
Survey 


Spectrum 


Analyzer 
GPS Environment 


Altitude 
Encoder Data 
Acquisition 


Display System 
Real-Time 


Post-Process Filter 
Smoothe 


Loosely/Tightly Integrated System 


Data Link 
Real-Time 


Post-Mission Processing 


Ground Transponders / Pseudolites / Track Space Time 31 


“Satellite Reference Station( SRS ) 


¥ 


Madified 


SimGEN 
PC 


Equivalent to a 


Equivalent to a 
separate GPS 
Simulator 


7 x GSS7700 
Dual RF 


separate Digital 
Jammer Generator 


Mon/Cal #2,5_| 


Mon/Cal #2, 


Jam In #1,6 


Mon/Cal #2 


Jam In #1,7 


Jam In #7,5© 


Wave Front CRPA Simulation S 


Master Unit Master Unit 


L1,2 (Jam) 
L1,1(GPS) 


L1,2 (Jam) 
RF Out#1,2 
L1,1(GPS) 


L1,2 (Jam) 


RF Out#1,3 
L1,1(GPS) 


L1,2 (Jam) 
RF Out#1,4 
L1,1(GPS) 


Signal Distribution 


Mon/Cal #2,1 
RF Out#1,1 


Jam In #1,1 


Mon/Cal #2, 


Jam In #1,2 


Mon/Cal #2, 


Jam In #1,3 


Mon/Cal #2,4 


Jam In #1,4 


L1,2 (Jam) 


RF Out#1,5 
L1,1 (GPS) 


a 
© | 11,2 (Jam) 


oe 


JU) SoOIDeTA PULAU 


RF Out#1,6 


L1,1(GPS) 
— 
7 | L1,2 Jam) 


RF Out#1,? 


—) L1,1 (GPS) 


Signal Distribution 


Backup Slide 
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23rd Annual NDIA 


National Test & Evaluation Conference 


Sustainment of the 
Deployed Navy Munitions Inventory 
through continuous Quality Evaluation Test 
and Evaluation 


March 14, 2007 


Jeffrey L. Johnson 

Director, Navy Quality Evaluation Program 
Naval Ordnance Safety and Security Activity 
Indian Head, MD 20640-5555 

Phone: 301-744-6049 

E-mail: jeffrey.johnson4@navy.mil 


hW\\7 He Naval Ordnance Safety and Security Activity 
AVAL SEA SYSTEMS COMMAND Quality Evaluation Program 


Ordnance Safety & Security Activity 


Provides an independent assessment of the continued safe 
and reliable performance of in-service weapons through 
routine and continuous testing 


Quality evaluation testing is the method used to predict and 
therefore prevent munitions failures that are inherent in 
their design or are the result of aging 


Provides a significant contribution in assuring that the Fleet 
is not the first point of detection of catastrophic failures 


Predicting failures and degraded performance is necessary 
to ensure that Fleet readiness can be assured throughout 


the reprocurement cycle 


NAVSEA Quality Evaluation Program Objectives 


NAVAL SEA“SYSTEMS COMMAND 


Ordnance Safety & Security Activity 


Assess the safety, reliability, & performance of the Navy 
weapons arsenal 


Identify and monitor critical characteristics that change 
with age and exposure 


Detect performance changes related to age and service use 


Predict future performance and service life/service 
limitations 


NAVSEA Quality Evaluation Program Domain 


NAVAL SEA“SYSTEMS COMMAND 


Ordnance Safety & Security Activity 


All in-service Navy weapons and ordnance programs 


In-service inventory - currently valued at $40B 


" Overall age of inventory is 15 — 20 years 


Um ‘g =A Life Cycle Test & Evaluation vs. 
NAVAL SEA“SYSTEMS oMMAND Life Cycle Cost 


Ordnance Safety & Security Activity 


T&E Governed by DoDINST Quality Evaluation Test & Evaluation Domain 


5000.2 


Life Cycle Cost 


operating & support cost 
investment 
development cost 


research & 
development cost 
disposal cost 


I j > 
concept SS operations & support oe disposal 


exploration! <— production & 
definition deployment 


demonstration/ 
Validation 


engineering/manufacturing development 


NAVSEA Quality Evaluation Program Contributions 


“NAVAL SEA SYSTEMS COMMAND 


Ordnance Safety & Security Activity 


Condition should drive maintenance 


Provides objective and independent test and evaluation of 
deployed munitions 


Identifies potential performance degradation during the 
operational phase 


Provides assurance that lethality, reliability and safety is retained 
in the operational phase 


Predicts the effect of system degradation of life cycle readiness 
and logistics issues 


Wr Life Cycle Readiness and Logistics Impact of 
NAVAL SEA‘SYSTEMS COMMAND Agin g an d Exposure 


Ordnance Safety & Security Activity 


Degradation of reliability and performance characteristics 
critically affect lethality and P, 


Inventory requirements depend on computational methods 
using as-built P, without regard for degradation of the 
deployed inventory 


Relying on as-built reliability and performance to determine 
the quantity of weapons needed to meet mission objectives 
overlooks the effect of degradation trends understates 
actual inventory requirements 


NAVSEA Typical War Fighter Concerns 


NAVAL SEA“SYSTEMS COMMAND 


Ordnance Safety & Security Activity 


Is normal use/consumption still possible without risk to the 
user after the ammunition is exposed to field conditions? 


Does returned ammunition have the same capability as 
“normal” ammunition? 


Are the minimum conditions under which ammunition is 
currently tested & evaluated during development sufficient 
for service use in an operational environment? 


WVe\W7TFs4 = Typical War Fighter Concerns (Continued) 


NAVAL SEA“SYSTEMS COMMAND 


Ordnance Safety & Security Activity 


Will it be necessary to define restrictions for operational 
use to ensure in-service safety? 


Does Ammunition Surveillance ensure unrestricted use, or 
should special field conditions and mission time be taken 
into consideration? 


Can undesirable changes to the ammunition be avoided by 
providing special storage conditions (air-conditioned 
containers or other storage modules)? 


nV \ | Traditional Aging Studies and 
NAVAL SEA SHSTENS COMMAND Predictive Analysis 


Ordnance Safety & Security Activity 


Traditional approach relied on destructive testing of 
meticulously selected test samples 


Limited sample sizes creates potential for statistical error 


Aging studies discover abnormalities caused by aging factors 


Studies provide basis for risk assessment and important 
retain/discard decisions 


Ni AVSEA Quality Evaluation Program Technical Desires 


Ordnance Safety & Security Activity 


Condition-based maintenance 

= Increased availability of life-cycle T & E information 
Health monitoring 

= Aging studies and predictive analysis 


= Modeling and simulation 
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ee 


NAVSEA Increased Availability of Life-cycle 


NAVAL SEA“SYSTEMS COMMAND 


Ordnance Safety & Security Activity T & IE | n fo rm ati Oo n 


= Operational testing and maintenance data 
©" Fleet training results including telemetry data 
= Improved and integrated data bases and data analysis tools 


= Zero-age performance data including the results of 
chemical and physical analysis 


12 


WV \ V7 7 Y Health Monitoring Approach 


NAVAL SEA“SYSTEMS COMMAND 


Ordnance Safety & Security Activity 


© Integrated maintenance and failure-reporting data systems 
= Adoption of standardized/adapted best-practices across systems 
" Failure and assessment modeling toolbox 


“= Development and deployment of micro-electro-mechanical (MEMS) 
sensors 


»" Remote collection of real-time location data and environmental data 


= Labor-free monitoring of environmental conditions 


© Determination of failure liability and the applicability of warranty 
coverage 
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Evolution of 
NAVAL SEA“SYSTEMS COMMAND Microelectromechanical (MEMS) Sensors 


Ordnance Safety & Security Activity 


Designed-in 


oe Built-in SSMD 
‘ 


) 


lines 


—= 2010 
and Beyond 


Goal: science based knowledge of 100% of the inventory, 
100% of the time, via 100% non-destructive means 


Maximized self-sensing and monitoring devices (SSMD) 


Increased use of non-destructive technologies and 
modeling/simulation techniques (100%) 


Reduced non Fleet-training destructive testing 


ee 


MV VA) uN Modeling and Simulation 


NAVAL SEA*‘SYSTEMS COMMAND 
Ordnance Safety & Security Activity 


™ Models are needed to assess reactions to multi-variables such 
as temperature, humidity, vibration, shock 


= Simulation of aging effect will permit understanding of resulting 
changes 


™ Understanding of changes will permit determination of 
remaining safe and useful service life 


NAVSEA Conclusions 


NAVAL SEA“SYSTEMS COMMAND 


Ordnance Safety & Security Activity 


Objective and independent life cycle test and evaluation of in-service 
weapons are necessary to assure weapons meet operator needs 
throughout service 


Maintenance and re-procurement plans should be based on how well 
the systems meet requirements 


Test-until-failure analysis is giving way to the analysis of degradation 
forces using aging and environmental exposure simulation models 


Real-time self-sensing monitoring systems will contribute to the 
quantification of aging effects and the reliable determination of 
remaining service life 


Standard M & S tools will be relied on for the assessment of the effects 
of degradation caused by aging and environmental exposure on the 
continued safety, performance and reliability of in-service munitions 
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NAVSEA OUR GOAL: FAILURE PREVENTION 


NAVAL SEA“SYSTEMS COMMAND 
Ordnance Safety & Security Activity 
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OUTLINE 


¢ About NUWC Division Keyport 
¢ About CTEC (Collaborative T&E Capability) 
¢ ESMS (Event Streaming Media System): 

= What is it & Why? 

= Some Details about the ESMS tool 

= ESMS Replay 


¢ Conclusion 
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NAVSEA CTEC 


ReYPORT (Collaborative Test & Evaluation Capability) 


SEA POWER 21 


~ Sea Shield 


Sea Basing — 


“,.. develop Sea Power 21 capabilities 
... by coordinating war gaming, experimentation & exercises 
... using technology push, concept pull, and spiral development 
... with the Fleet at the heart of innovation. ” 
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NAVSEA CTEC 
warmers —  (COllaborative Test & Evaluation Capability) 


KEYPORT 
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Focus Area: The Undersea / Littoral Warfare Component 
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¢ CTEC, the Capability, is comprised of 
three complementary components: 


¢ Facility 
¢ Process 


Fort Irwin 


~ 


Joint Network 
(GCCS, TENA, ...) 


_ Nellis AFB 


| PURE 
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EC: Facility, Tools, & Process 
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A Event Streaming Media System (ESMS) Concept 


WARFARE CENTERS 


KEYPORT 


Stream 
Chose Adichat.. | Dail. | Review Fauci -Range Comms Balet Ba] 
—EE 


r Mite —} 
Notifications Range Co. | EE 2) oS Goon 
ae “— : A 
—— Publish Client: 


Sy : IE w/Flash Player 
ee oe Publish/Present 
‘ Graphical User 

10:21:04 New Chat - Range Safety | nte rface 


10:22:27 New Chat - Acoustic Analysis 


10 [10:20:52 Person1: Message 1 

10:21:04 Person2: Message 2 : rr -Live- 

tbecdlead heer lever ESMS - Event AL2004 - Live - UNCLASSIFIED 
Event Stream Window Timing 


[ms GCCS Display lolx) CSUMIRueis PEEVES] submarine Detection Range Display file! B4| Seivers: 
Adobe Flash Comm 


Microsoft SQL 


"| [My Message 


el Pp ia [1 —— 
| Status 


UAV Camera ft EalelE SH6OSONARLOFAR life EA 


Chat - Range Comms. _—/olx! Z a? : ; = PC-4MAT Display 

pels soo ——_—— 

Local: [ Mute ——= -—————— Presentat Client: 
200 821 RCO: S80 SONAR dptyes IE w/Flash Player 


Z09:22:36 ASSESS1: Threat Sub detected J 
Z09:35;18 RCO: SH-60 SONAR retrieved 
x 


| SH-60 SONAR retrieved Send 


mb | aa oe Te 


‘ , ‘ ' , ' ‘ ' ‘ 
| Event Time: 209:35:42 
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NAVSEA Why ESMS Is Needed? 


WARFARE CENTERS 


KEYPORT 


Desired Features 


Total 
Feature Set Features 


e Incomplete and/or inaccessible archive => limited cost anes ee eel 
4 5 Event Administration Management 
effective test reconstruction 
¢ Limited live event visibility => lower test productivity 


¢ Diverse event viewing roles & requirements 
e Extended duration events with ad hoc participation 


: . . - - Offline E Timing M 5 

- Complex event viewing & analysis applications cee vies eed 

¢ Commercial media server products are not optimized for 
event insight 


Satan nUcincce event Insight Group Administration Management Eo 
Characteristic | Communication (ESMS) 


Viiesy | ailenes) 


e Limited event data correlation => limited insight 
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4 (lowest) 1 (highest) 


KEYPORT 


4 


ESMS Display 


Keyport Recovery Vehicle 


System Acceptance and Operational Readiness Department 
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ESMS Fly-Away Kit 
ESMS Features 


e Synchronous information from distributed 
information sources (audio, video, screen 
capture) for a single event 


e Correlated analysis insight in a distributed 
environment 


e Replay/review capability (“TIVO” like) 

e Configurable perspectives supporting 
insight for many job functions 

e Distributed collaboration environment 

e Event comparison through live archive 


access query * Panasonic Rugged notebooks 
. i * PTZ Video Cameras 
e Continuous event life cycle through re- * Lorex Infrared Cameras 
: - ¢ Epiphan Frame Grabber 
publication * Osprey 440 Video Capture Card 
- 4 : * AN Cables 
e Distributed server design to enhance * Linksys Wireless Router & Hub 
: . ¢ Harris Wireless Bridge for SECNET-11 
communication performance * Adobe Flash Media Server 2 


e UPS Battery Backup 


System Acceptance and Operational Readiness Department 
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The Critical Link Between Test & Evaluation 
and Modeling & Simulation 


Dr. Mark J. Kiemele 
Air Academy Associates 


A Tutorial for NDIA’s 
23" Annual National Test and Evaluation Conference 
Hilton Head Island, SC 
March 12, 2007 
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Introductions 


= Name 

= Organization 

= Job Title/Duties 

= Experience in DOE, T&E, M&S 


Simplify and Perfect 
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Agenda 


= Some Basic Definitions 


Test and Evaluation (Experimental) Approaches 
= DOE: the Gateway to Transfer Functions and Optimization 
= Break 


= Monte Carlo Simulation, Robust Design, and Tolerance 
Allocation 


= Examples of Iterative Use of Modeling and Simulation 


= Summary of “Modeling the Simulator” 


Simplify and Perfect 
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Some Basic Definitions 


=" System: a collection of entities which act and interact together to 
achieve some qoal 


= Model: a simplified representation of a system developed for the 
purpose of studying a system 


= Simulation: the manipulation of a model in such a way that it allows 
the investigation of the performance of a system 


= Modeling and Simulation: a discipline for developing a level of 
understanding of the interaction of the parts of a system, and of the 


system as a whole 


Simplify and Perfect 
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About Models 


All models are simplifications of reality. 


There is always a tradeoff as to what level of detail should be 
included in the model: 


If too little detail, there is a risk of missing relevant 
interactions and the resultant model does not promote 
understanding. 


If too much detail, there is a risk of overly complicating the 
model and actually preclude the development of 
understanding. 


The “goodness” of a model depends on the extent to which it 
promotes understanding. 


Simplify and Perfect 
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Simplify and Perfect 


Types of Models 


High-Fidelity Models: 
- many variables and many interactions 
- highly detailed and complex 
- needed for visualization 
- difficult to manipulate 


Low-Fidelity Models: 
- much fewer number of variables 
- can be manipulated more easily 
- provides higher-level view of system 
- presents a more aggregate view of the 
system 
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Types of Simulation 


Deterministic Simulation: 
- for each combination of inputs parameters, 
there is one and only one output value 


- y= A(x) 


Monte Carlo Simulation: 
- provides for variability in the inputs 
- y = f(x + variation), where the variation is 
modeled as some probability distribution 


Discrete Event Simulation: 
- studies a sequence of countable events 
- assumption is that nothing of importance 
takes place between events 


Simplify and Perfect 
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Definition of a Process 


A blending of 
inputs to 
achieve some 
desired outputs 


Simplify and Perfect 
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Graphical Meaning of y and o 


y= Average = Mean = Balance Point 


o = Standard Deviation 


Concave Down 


Inflection Point 
o = 160 — 153= 7 


o = average distance of points 
from the centerline 


Simplify and Perfect 
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Graphical View of Variation 


+30: Natural Tolerances 


99.9937% ! 
99.999943% 
99.9999998% 


Typical Areas under the Normal Curve 
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Experimental Methods 


¢ Traditional Approaches 
¢ One Factor at a Time (OFAT) 
¢ Oracle (Best Guess) 


¢ All possible combinations (full factorial) 


¢ Modern Approach 


¢ Statistically designed experiments (DOE) ... full 
factorial plus other selected DOE designs, 
depending on the situation 


Simplify and Perfect 
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Simplify and Perfect 


OFAT (One Factor at a Time) 


X,= time a 
. 
Chemical Reaction Yield 
— ele 
X, = temp Process 
[ae 


One factor at a time results versus optimal results 


60 90 120 150 180 
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Oracle (Best Guess) 


Run 


CONOOaRWHNDN = 
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Evaluating the Effects of Variables on Y 


What we have is: 


What we need is a design 
to provide independent 
estimates of effects: 


©) 


Simplify and Perfect How do we obtain this independence of variables? 
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Simplify and Perfect 


Example 1: 


A (2 levels) 
B (2 levels) 


Example 2: 


A (3 levels) 
B (3 levels) 
C (@ levels) 


All Possible Combinations 
(Full Factorial) 


MATRIX FORM 


NN-=-4 


> 


WHOWNNNSS2SWWHWNNNAAA 


NSN 


WNSBWNABWNABWNAWNAWNA 
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TREE DIAGRAM 
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What Is a Designed Experiment? 


Purposeful changes of the inputs (factors) in order to observe 
corresponding changes in the output (response). 


Aig Y 
a —___1____+ 
—_____» 
Inputs =x, PROCESS Y, Outputs 
—+_______» a a 
x , 
—________» 


Simplify and Perfect 
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i) 


ii) 


Iv) 


Simplify and Perfect 


DOE Helps Determine How Inputs Affect 


Outputs 


Factor A affects the average of y 


4 ; XN 


y 
Factor B affects the standard deviation of y 


B, 
B, 


Factor C affects the average and the 
standard deviation of y 


C, 
C, 


Factor D has no effect on y 


D,=D, 


y 
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Transfer Functions 


Parameters 
or Factors y (CTC) 
that 
Influence 
the CTC 
Where does the transfer function come from? 
¢ Exact transfer Function 
¢ Approximations 
- DOE 
- Historical Data Analysis 
Simplify and Perfect - Simulation 
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Exact Transfer Function 


¢ Engineering Relationships 
- Ve=IR 
- F=ma 
Ry 


The equation for the impedance (Z) 
fo through this circuit is defined by: 
Fs R, : R, 


R> R,+R 


1 2 


The equation for magnetic force at a distance 
X from the center of a solenoid is: 


: ’ “| 5l+x 5l—x 


—-— a —— 
20) Jr? +(.52+xy fr? +(.52-—xy 


4 


Where : total number of turns of wire in the solenoid 
: Current in the wire, in amperes 
: radius of helix (solenoid), in cm 
length of the helix (solenoid), in cm 
: distance from center of helix (solenoid), in cm 


> magnetizing force, in amperes per centimeter 


Ix. Sa Zz 


Simplify and Perfect 
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Catapulting Power into Design of Experiments 


Art 
eee re Statapult® Catapult 
Simplify and Perfect 


The Theoretical Approach 


Simplify and Perfect 
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The Theoretical Approach (cont.) 


- 7 _ Dt siné 
|,98 = r-F(8) sin6cos@ — (Mgr, + mgr, ) sin® cL ererareey er, 
s) 
; L, 6? = al F(8)sin8 cos@dé — (Mgr, +mgr, )(sin® — sin8, ) 
Q% 
; 0; 
; |, 02 = - | F(@)sine cos gdé@ — (Mgr, + mgr, )(sin8, — sin8, ). 
Q 
Tl ; er, iE 1 
K=_V, cos( 7 - 0} 5" cos 8, y =r, sin®, + V, sin( 5 = 6} ~5at'. 


2 
fg sind, +(R+tp c0804)tan| £04] __9 (R+tp cos 01)" _ 0. 


2 
2VB cos? E - a) 


Glo (R+1p Cos 0, a 
4 ' 
B cos? (é - 01) t sin 8, +(R-+ tp cos 6, tan{ 2 - )| 
6 
= | F(8)sin6 cos ¢d0 —(Mgrg + mgr )(Sin 84 — sinQg). 
Simplify and Perfect & 
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Statapult® DOE Demo 


(The Empirical Approach) 


Actual 
Factors 


Coded Factors Response Values 


Simplify and Perfect 
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The Beauty of Orthogonality 


A Full Factorial Design for 3 Factors A, B, and C, Each at 2 levels: 


Run A B Cc AB AC BC ABC 


1 - = - + + + - 
2 - 7 + + “ = + 
3 - + - - + 7 + 
4 - + + - - + - 
5 + - - - - + + 
6 + a + 7 + - a 
7 + + - + - - - 
8 + + + + + + + 


Simplify and Perfect 
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Value Delivery: Reducing Time to Market for 
New Technologies 


INPUT OUTPUT 


Pitch <) (0, 15, 30) 


Roll < (0, 15, 30) 


a 


W1EF <) (-15, 0, 15) 


2 | 


W2F <) (-15, 0, 15) 


| 


W3F <) (-15, 0, 15) 


Modeling Flight 


Characteristics 


Six Aero- 


of New 3-Wing 
Aircraft 


Characteristics 


¢ Total # of Combinations = 3° = 243 
e Central Composite Design: n = 30 


Siolieaciesiceas Patent Holder: Dr. Bert Silich 
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Aircraft Equations 


C.=  .233 + .008(P)? + .255(P) + .012(R) - .043(WD1) - .117(WD2) + .185(WD3) + .010(P)(WD3) - 
.042(R)(WD1) + .035(R)(WD2) + .016(R)(WD3) + .010(P)(R) - .003(WD1)(WD2) - 
.006(WD1)(WD3) 


Cp=  .058 + .016(P)? + .028(P) - .004(WD1) - .013(WD2) + .013(WD3) + .002(P)(R) - .004(P)(WD1) 
- .009(P)(WD2) + .016(P)(WD3) - .004(R)(WD1) + .003(R)(WD2) + .020(WD1) + .017(WD2)2 
+ .021(WD3)? 


Cy=  -.006(P) - .006(R) + .169(WD1) - .121(WD2) - .063(WD3) - .004(P)(R) + .008(P)(WD1) - 
.006(P)(WD2) - .008(P)(WD3) - .012(R)(WD1) - .029(R)(WD2) + .048(R)(WD3) - .008(WD1)2 


Cy=  .023 - .008(P)2 + .004(P) - .007(R) + .024(WD1) + .066(WD2) - .099(WD3) - .006(P)(R) + 
.002(P)(WD2) - .005(P)(WD3) + .023(R)(WD1) - .019(R)(WD2) - .007(R)(WD3) + .007(WD1)2 
- .008(WD2)? + .002(WD1)(WD2) + .002(WD1)(WD3) 


Cyy=  .001(P) + .001(R) - .050(WD1) + .029(WD2) + .012(WD3) + .001(P)(R) - .005(P)(WD1) - 
.004(P)(WD2) - .004(P)(WD3) + .003(R)(WD1) + .008(R)(WD2) - .013(R)(WD3) + .004(WD1)? 
+ .003(WD2)? - .005(WD3)2 


C,=  .003(P) + .035(WD1) + .048(WD2) + .051(WD3) - .003(R)(WD3) + .003(P)(R) - .005(P)(WD1) 
+ .005(P)(WD2) + .006(P)(WD3) + .002(R)(WD1) 


Simplify and Perfect 
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Hierarchical Transfer Functions 


; Gross Profit 
Y = Gross Margin = 
Gross Revenue 


Y= fly, Yo ¥3> Yao Y55 Ye) 


yi Y2 Y3 
= (Rev - COG) + (Rev 


Y4 
— Cost 


equip post sales post sales) 


Y4 
Cost 


X, Xo X3 


post sales = f(field cost, remote services, suppliers) 


x, = f(direct labor, freight, parts, depreciation) 


Simplify and Perfect 


r 
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Modeling The Drivers of Turnover 


@ External Market Factors 
(Local Labor Market Conditions) 


Local Unemployment Rate 
Local Employment Alternatives 
— Se ——_—  o — 


Company’s Market Share Process of 


@ Organizational Characteristics Deciding to Turnover Rate 
and Practices 


Stay / Leave 


Supervisor Stability 
Lateral / Upward Mobility 
Layoff Climate 


@ Employee Attributes 
Time Since Last Promotion 
Education Level 
Job Stability History 


Simplify and Perfect 
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DOE “Market Research” Example 


Suppose that, in the auto industry, we would like to investigate the following automobile 
attributes (i.e., factors), along with accompanying levels of those attributes: 


A: Brand of Auto: -1 = foreign +1 = domestic 

B: Auto Color: -1 = light 0 = bright +1 = dark 

C:Body Style: -1 = 2-door 0 = 4-door +1 = sliding door/hatchback 
D:Drive Mechanism: _ -1 = rear wheel 0 = front wheel +1 = 4-wheel 

E: Engine Size: -1 = 4-cylinder 0 = 6-cylinder +1 = 8-cylinder 

F: Interior Size: -1 <2 people 0 = 3-5 people +1>6 people 

G: Gas Mileage: -1 < 20 mpg 0 = 20-30 mpg +1>30 mpg 

H:Price: -1 < $20K 0 = $20-$40K +1 >$40K 


In addition, suppose the respondents chosen to provide their preferences to product 
profiles are taken based on the following demographic: 


J: Age: -1 < 25 years old +1 > 35 years old 
K: Income: -1 < $30K +1> $40K 
L: Education: -1<BS +1> BS 


Simplify and Perfect 
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DOE “Market Research” Example (cont. ) 


Question: Choose the best design for evaluating this scenario 


Answer: L,, design with attributes A - H in the inner array and 
factors J, K, and L in the outer array, resembling an 
L,, robust design, as shown below: 


+ + 
AB CDEFGH| [Wuw uw Hw [o/s 


Segmentation of the population or 
Respondent Profiles 
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OONDARWDND = 


| 


t+t000+118148 


Simplify and Perfect 


* 18 different product profiles 


Multiple Response Optimization Simulation 
Example 


Cross Support 


Diameter (2-4) Design Cost (< 300) 


Vertical Support 


Diameter (2-4) Vehicle 
Impact Rib Compression (< 11) 
ba 
Safety 
Box Diameter (2-4) Process 


Triaxial Acceleration (< 15) 
ee 


Beam Angle (35-60) 


Simplify and Perfect 
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Optimization Using Monte Carlo Simulation 


¢ Expected Value Analysis 
¢ Robust Design 


¢ Tolerance Allocation 


Simplify and Perfect 
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Expected Value Analysis (EVA) 


Previously, we discussed variability in the outputs in great detail. 
However, the inputs also have variability. 


y,~ N(50, 3) S\N 


If a product has never been produced, how do we know what the 
distribution of the output is? What is the expected value and 
variance of y? 


Simplify and Perfect 


F 
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Expected Value Analysis Example 


ne y = x? 


¢ What is the shape of the output (y) distribution? 
¢ What is the mean or expected value of the y distribution? 


¢ What is the standard deviation of the y distribution? 


Simplify and Perfect 
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Robust Design 


7. 
ra 


Process of finding the optimal location parameters (i.e., means) 
of the input variables to minimize dpm. 


Simplify and Perfect 
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Why Robust Design? 


One Input 


If u, varies, should we select p, or p, to hit y = T? 


Simplify and Perfect 
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Simplify and Perfect 


Ball Valve Pressure (100-200) 
—_— Os 


Robust (Parameter) Design 
Simulation Example 


Plug Pressure (20-50) 


Bellow Pressure (10-20) 


Water Temp (70-100) 
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Nuclear 


Reservoir 
Level Reservoir Level (700-900) 
pe =e 


Control 
Process 
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Tolerance Allocation (TA) 


The process of quantifying the sensitivity of the output (y) dpm 
to changes in the input variables’ (X's) standard deviations. It 
provides the designer the ability to perform cost/benefit tradeoffs 
Wieiiecs Via assignment of standard deviations to the input variables. 
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Tolerance Allocation Example 


R, ~ N (50,22) 


Impedance 


R,~ N (100,22 ) Example 


If we were able to change a resistor’s standard deviation, which 
resistor, R, or R,, would have the greater impact on the dpm of Z 
(impedance)? 


Simplify and Perfect 
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Tolerance Allocation Example (cont.) 


A reduction of R, by 50% reduces dpm by an order of magnitude 
X, while R, has little impact. 


We) (-Ye-laler-w_eN Coker: 1d(e) aim K-10) (=) 
N = 10,000 (in defects per million) 


Taal exexer-TaCer-mm ir-le) (= 


-50% Sigma 372.40 34,683 


-25% Sigma 8,058 36,849 
-10% Sigma 23,906 35,663 
Nominal 39,220 39,657 
+10% Sigma 59,508 37,556 
+25% Sigma 92,398 47,317 
+50% Sigma 148,113 46,801 


A reduction of R,'s standard deviation by 50% combined with an 
increase in R,'s standard deviation by 50% 


R, ~ N(50, 1°) 


R, ~ N(100, 3°) 


Simplify and Perfect 


results ina dpm = 1,254. 
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Examples of Simulation and High 
Performance Computing (HPC) 


Power 


Simulation of stress and vibrations of turbine 
assembly for use in nuclear power generation 


Automotive 


Simulation of underhood thermal cooling for decrease 
in engine space and increase in cabin space and comfort 
Aerospace 
Evaluation of dual bird-strike on aircraft engine 
nacelle for turbine blade containment studies 
Electronics 
Evaluation of cooling air flow behavior 
inside a computer system chassis 
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Simplify and Perfect 


Examples of Computer Aided Engineering 
(CAE) and Simulation Software 


Mechanical motion: Multibody kinetics and dynamics 
ADAMS® 
DADS 


Implicit Finite Element Analysis: Linear and nonlinear statics, 
dynamic response 

MSC.Nastran™, MSC.Marc™ 

ANSYS® 

Pro MECHANICA 

ABAQUS® Standard and Explicit 

ADINA 


Explicit Finite Element Analysis : Impact simulation, metal 
forming 

LS-DYNA 

RADIOSS 

PAM-CRASH®, PAM-STAMP 


General Computational Fluid Dynamics: Internal and external 
flow simulation 

STAR-CD 

CFX-4, CFX-5 

FLUENT®, FIDAP™ 

PowerFLOW® 
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Simplify and Perfect 


Simplify and Perfect 


Examples of Computer Aided Engineering 
(CAE) and Simulation Software (cont. ) 


Preprocessing: Finite Element Analysis and 
Computational Fluid Dynamics mesh generation 

ICEM-CFD 

Gridgen 

Altair® HyperMesh® 

Il-deas® 

MSC.Patran 

TrueGrid® 

GridPro 

FEMB 

ANSA 


Postprocessing: Finite Element Analysis and 
Computational Fluid Dynamics results visualization 

Altair® HyperMesh® 

Il-deas 

MSC.Patran 

FEMB 

EnSight 

FIELDVIEW 

ICEM CFD Visual3 2.0 (PVS) 

COVISE 
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Multidisciplinary Design Optimization (MDO).: 
A Design Process Application 


Design Responses (Minimize) 


Discipline 2 
Optimum 


Multidis ciplinary 
Optimal Design 


Key Elements of MDO 


>< Discipline 1 * Massive Computational Problem; 
Optimum * Solution by decomposition effective for complex systems; 
* Multiprocessor computing simplifies MDO solutions 
conceptually & enables solutions previously intractable; 
* Aids in the management of the design process. 


eens Feasible design space (discipline 1) 
Feasible design space (discipline 2) 


Design Variables 


Mastery of interactions between the disciplines (or, subsystems) 
is as important as the methods & tools used within a single discipline 


Simplify and Perfect 
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Latin Hypercube Sampling 


Method to populate the design space when 
using deterministic simulation models. 


Design space has k variables (or dimensions). 
Ex: Assume k = 2 

Suppose a sample of size n is to be taken; "2 
Stratify the design space into nk cells. 

Ex: Assume n = 5; nk = 52 = 25 

Note: there are n strata for each of the k 
dimensions. 


x4 
Each of the n points is sampled such that each 
marginal strata is represented only once in the 
sample. 


Note: each sample point has its own unique 
row and column. 


Xy 


Simplify and Perfect 
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Applying Modeling and Simulation to 
Automotive Vehicle Design 


IDENTIFY | _“2y. Mary's | SCREENING DESIGN |_ The critical 
CTCs, CDPs (DOE PRO) cis 


Examples of CTCs: NASTRAN RADIOSS = MADYMO 
y, = weight of vehicle Integrated processes with high fidelity 
CAE analyses on HPC servers 
Y> = cost of vehicle 


y3 = frontal head impact Examples of Critical Design Parameters | Examples of Critical Design Parameters (CDPs or Xs):_| or Xs): 


Safety CTCs ett oak ' 
with constraints | ¥4 ~ ontal chest impac 
specified by Ys = toe board intrusion dee x, = roof panel material 
= 
FMVSS Yg = hip deflection pale ; 
(Federal Motor ae MADYMO 4g = Toor pane Mhickhess 
Vehicle Safety ce Veer nne x3 = door pillar dimensions => i beam 
Standards) Yg = side impact 


X,4 = shape/geometry 


= int | d i irfl 
Yq = internal aerodynamics (airflow) | c= Windshield Glass 


= external aerodynamics (airflow 
v6 2 ( ) Xg = hood material, sizing and thickness 


no federal Y4, = noise 
requirements 7 ae X7 = under hood panel material, sizing and thickness 
on these CTCs Y42 = vibration (e.g., steering wheel) NASTRAN 


Y13 = harshness (e.g., over bumps, shocks) 
Y14 = durability (at 100K miles) 


Simplify and Perfect 


©2007 Air Academy Associates, LLC. Do Not Reproduce. Page 45 


Applying Modeling and Simulation 
to Automotive Vehicle Design (cont. ) 


Robust 
CDPs Designs 
——> ——__——> 


sa CTCs 


/NASTRAN RADIOS MADYMO RADIOSS MADYMO /NASTRAN RADIOSS MADYMO]| Response Surface Models Surface Models [NASTRAN RADIOS MADYMO [NASTRAN RADIOS. MADYMO RADIOSS MADYMO 


Simplify and Perfect 
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Introduction to High Throughput 
Testing (HTT) 


¢ Arecently developed technique based on combinatorics 


¢ Used to test myriad combinations of many factors (typically qualitative) 
where the factors could have many levels 


« Uses a minimum number of runs or combinations to do this 


¢ Software (e.g., ProTest) is needed to select the minimal subset of all 
possible combinations to be tested so that all n-way combinations are tested. 


¢ HTT is not a DOE technique, although the terminology is similar 


¢ Arunorrow in anHIT matrix is, like DOE, a combination of different factor 
levels which, after being tested, will result in a successful or failed run 


¢ HTT has its origins in the pharmaceutical business where in drug discovery 
many chemical compounds are combined together (combinatorial chemistry) 
at many different strengths to try to produce a reaction. 


¢ Other industries are now using HTT, e.g., software testing, materials 
discovery, interoperability testing, IT (see IT example on next page) 
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Simplify and Perfect 


HTT Example 


¢ An IT function in a company wanted to test all 2-way combinations of a 
variety of computer configuration-related options or levels to see if they 
would function properly together. 


¢ Here are the factors with each of their options: 
Motherboards (5) : Gateway, ASUS, Micronics, Dell, Compaq 


RAM (3) : 128 MB, 256 MB, 512 MB 

BIOS (3) : Dell, Award, Generic 

CD (3) : Generic, Teac, Sony 

Monitor (5) : Viewsonic, Sony, KDS, NEC, Generic 
Printer (3) : HP, Lexmark, Cannon 

Voltage (2) > 220, 110 


Resolution (2) : 800x600, 1024x768 
¢ How many total combinations are there? 


¢ What is the minimum number of these combinations we will have to test 
(and which ones are they) in order to determine if every 2-way combination 
(e.g., Dell Bios with Teac CD) will indeed work properly together? 

¢ To answer this question, we used Pro-Test software. The answer is 25 
runs and those 25 combinations are shown on the next page. 
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Simplify and Perfect 


Simplify and Perfect 


High Throughput Testing (HTT) 


(for all two-way combinations) 


Case 1 
Case 2 
Case 3 
Case 4 
Case 5 
Case 6 
Case 7 
Case 8 
Case 9 
Case 10 
Case 11 
Case 12 
Case 13 
Case 14 
Case 15 
Case 16 
Case 17 
Case 18 
Case 19 
Case 20 
Case 21 
Case 22 
Case 23 
Case 24 
Case 25 


5 Levels 
Motherboard 
ASUS 
Compaq 
Gateway 
Dell 
Micronics 
Gateway 
Micronics 
ASUS 
Compaq 
Micronics 
Dell 
Gateway 
Compaq 
ASUS 
Micronics 
Gateway 
Dell 
ASUS 
Compaq 
Gateway 
Micronics 
ASUS 
Dell 
Compaq 
Dell 
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3 Levels 3 Levels 


RAM 

256 MB 
512 MB 
128 MB 
128 MB 
256 MB 
256 MB 
512 MB 
512 MB 
128 MB 
512 MB 
256 MB 
512 MB 
256 MB 
128 MB 
128 MB 
128 MB 
128 MB 
256 MB 
512 MB 
256 MB 
512 MB 
256 MB 
512 MB 
256 MB 
128 MB 


BIOS 
Dell 
Dell 
Generic 
Award 
Generic 
Award 
Award 
Generic 
Award 
Generic 
Award 
Dell 
Generic 
Dell 
Dell 
Generic 
Dell 
Award 
Dell 
Generic 
Generic 
Generic 
Generic 
Dell 
Generic 


Do Not Reproduce. 


3 Levels 
CD 
Generic 
Teac 
Sony 
Teac 
Teac 
Sony 
Generic 
Teac 
Generic 
Teac 
Generic 
Sony 
Sony 
Sony 
Sony 
Teac 
Sony 
Sony 
Sony 
Generic 
Teac 
Generic 
Sony 
Generic 
Sony 


5 Levels 
Monitor 
Viewsonic 
Sony 
KDS 

NEC 
Generic 
Sony 
Viewsonic 
KDS 
Generic 
Sony 
KDS 
Generic 
Viewsonic 
NEC 

KDS 
Viewsonic 
Sony 
Generic 
NEC 

NEC 

NEC 
Sony 
Viewsonic 
KDS 
Generic 


3 Levels 
Printer 
Lexmark 
HP 
Cannon 
Cannon 
Lexmark 
HP 
Cannon 
HP 
HP 
Lexmark 
Lexmark 
Lexmark 
Cannon 
Cannon 
Lexmark 
HP 
Cannon 
Cannon 
Lexmark 
Cannon 
HP 
HP 
HP 
Cannon 
HP 
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2 Levels 2 Levels 


Voltage 
110 V 
220 V 
220 V 
110 V 
220 V 
110 V 
220 V 
220 V 
110 V 
110 V 
110 V 
110 V 
220 V 
220 V 
220 V 
110 V 
110 V 
220 V 
110 V 
220 V 
220 V 
110 V 
220 V 
220 V 
110 V 


Resolution 
800 x 600 
1024 x 768 
800 x 600 
1024 x 768 
1024 x 768 
1024 x 768 
1024 x 768 
1024 x 768 
800 x 600 
800 x 600 
1024 x 768 
1024 x 768 
1024 x 768 
800 x 600 
800 x 600 
800 x 600 
1024 x 768 
1024 x 768 
800 x 600 
800 x 600 
800 x 600 
800 x 600 
1024 x 768 
1024 x 768 
800 x 600 


Example of Iterative Approach to Modeling and 
Simulation to Optimize Transmission Performance 


Predicted Selected 


29 x's Quadratic DFSS 
Response Master: 5 Potential Cos in Best Go To 
Surface on 29 Robust Designs Presence | Starting Phase 2 
Variables Design of Noise Design 


A 


I 


: High Fidelity Monte Carlo Simulation 
a ies Model with various weightings 25-Level LHS Also, reduced to 
of responses inserted Noise in 11 Design 
Blue : Low Fidelity each of 8 Noise Variables plus 8 
= Variables and 21 Noise Variables 


Analytical Model ; 
Design Variables 


LHS: Latin Hypercube Sampling — a method for (testing the 
sampling a design space when deterministic designs in the 
simulation is being used in high dimensions, presence of noise) 


involving many input variables. 


LHS used to generate 5-rep outer 
array from 8 Noise Variables 


DOE Pro 
11 Design Var Quadratic Model Multiple 5 Potential Optimal 
8 Noise Var. Response Designs Design 
§ model Optimization 
t 25-Level LHS inserted Build and 
CCD Inner Array Noise in each of 8 Test 
Alters 11 Design Noise Variables and 21 Prototype 
Variables Design Variables 


(testing the designs in 
the presence of noise) 


Simplify and Perfect Celebrate 


Used with permission from John Deere 
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Technologies Used on Transmission Project 


- High Throughput Testing (HTT) 


To generate a minimal number of test cases; in this scenario, 11 
combinations of 29 variables that would allow testing all two-way 
combinations on Easy 5. This made running the Easy 5 simulator 
much easier, without interruption. 


- DFSS Master: for Robust Design and Expected 
Value Analysis (Monte Carlo Techniques) 


¢- Highly Orthogonal Latin Hypercube Sampling 


To populate the design space with test cases which are highly 
orthogonal. Typically used with deterministic simulation to screen out 
the CDPs and also to use modeling DOEs on the simulator to generate 
transfer functions which characterize the simulator 


¢ DOE PRO: for Multiple Response Optimization 


Simplify and Perfect 


Summary of "Modeling the Simulator" 


Critical Parameters ID'd 


Transfer Function on 
Critical Parameters 


Perform Expected Value Analysis, 
Robust Design, and Tolerance 
Allocation Using Transfer Function — 


Optimized Design 


Optimized Simulator 


Build Prototype to Validate 
Simplify and Perfect Design in Real World 


©2007 Air Academy Associates, LLC. Do Not Reproduce. Page 52 


Environments Where Simulation and Modeling 
ls Beneficial 


¢ A high number of design variables 
¢ A substantial number of design subsystems and 


engineering disciplines 


Interdependency and interaction between the subsystems 
and variables 


Multiple response variables 


Need to characterize the system at a higher level of 


abstraction 


Time and/or space must be compressed 


Simplify and Perfect 
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For Further Information, Please Contact 


Mark J. Kiemele 


Air Academy Associates, LLC 
1650 Telstar Drive, Ste 110 
Colorado Springs, CO 80920 


Toll Free: (800) 748-1277 or (719) 531-0777 
Facsimile: (719) 531-0778 
Email: aapa@airacad.com 
Website: www.airacad.com 


Mead 


ACADEMY 
Simplify and Perfect ASSOCIATES 
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P 


NDIA Test & Evaluation 
Conference 


Producing Anywhere, Anytime Test, 
Evaluation and Diagnostics Capable 
Products to Eliminate the T&E 
Logistic Burden 


Ryan Kinney 

Fault Detective Product Manager 
Ryan_Kinney@agilent.com 

(970) 679-5881 


Anywhere, Anytime Diagnostics 


-$°3- Agilent Technologies Agilent Restricted 
- March 14, 2007 


Introduction 


What is Agilent? 


e Although Agilent sells a diverse number of products, my part of the 
company designs, manufactures, and sells extremely complicated & 


state-of-the-art RF instruments tt Agilent Technologies 


¢ Even though we are an industry leader and our core competency is test 
& measurement, we struggle with “sustainment” 


Who am I? 


¢ | am the product manager for an Agilent software product named Fault 
Detective 


¢ My background is software development 


ee Agilent Technologies 


Anywhere, Anytime Diagnostics 
Agilent Restricted 


March 14, 2007 


What is “Anywhere, Anytime Diagnostics”? 


Anywhere Anytime 
A single diagnostics technology used across the entire Fast, accurate diagnostics regardless of location and 
product lifecycle operator skill 


1. Manufacturing/production 
2. Service centers/depots 
3. Inthe field (Both remote and local) 


Agilent Vision PAV IDAYAT=i Cela) 


am ©] a{- me) melUlalarsjigelealoJalicmr-tlicme)al A failure occurs on a jet in mid- 
Weleicice)aalol mcm parclaleireellelaiale vilfe lait 


line... halting production The jet self-diagnoses and 


¢ The instrument self-diagnoses oXe)anlanlelalierclictomualom reli (0lgomlemr| 
and notifies an Agilent service fol gelulalemeneaiy 


ollie The ground crew is prepared by 


¢ An Agilent repair technician the time the jet lands 
contacts the customer knowing 
exactly how to fix the problem 


Anywhere, Anytime Diagnostics 
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“Anywhere, Anytime Diagnostics” in Action 


Two Agilent instruments are equipped with “Anywhere, Anytime Diagnostics”: 
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Challenges We Faced 


1. Design a product with adequate diagnosability 
If a product is too difficult to repair, any sustainment strategy will be a challenge. Typically, 
diagnosability is not known until production, when it’s often too late to fix the problem. 


2. Create atest plan with appropriate coverage & diagnostic resolution 
Poor test coverage puts faulty products in the field. Poor diagnostic resolution produces test 
results that provide little insight into failures. Quantifying the effectiveness of functional test is 
notoriously difficult, making it hard to know when your test process is “good enough”. 


3. Automate trouble-shooting 
A worthy goal, but difficult to achieve without a lot of investment. 


4. Embed the entire solution in the product 
How do you create an effective test solution without any external test equipment? 
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Agilent Fault Detective Software 


Q: The issues on the previous slide are really tough to 
address, so how did we do it? 


A: We used an in-house software tool named Fault Detective. 


Agilent Fault Detective 


Take Control of Diagnosis and Repair! 


te Agilent Technologies 


© Agilent Technologies 1996 - 2006 
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Process Overview 


Product Design 


BIT Test Plan 


Fault Detective 


Process Overview 


Create/capture test plan in Fault Detective 
Analyze the test plan in Fault Detective to generate metrics 
USM aleitglexcmmcommian] 6)ae) om cele le (eime|(-lelareyst-le) ita 


Use metrics to improve test coverage and diagnostics resolution 
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Creating/Documenting Your Test Plan 
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Step 1 
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Step 2 
List hardware components 
(nouns) 


4 JY ob yjuaWUBIYIAModO7 


iyo 
4 Juyjuo0b 


4 

4 

4 

4 wy ob 

4 wy ob 

4 wy ob 
Uy 


4 
4 
4 
4 
4 
4 


Step 3 
List functionality (verbs) of each 
hardware component 


Step 4 

For each function, assign “amount 
of component exercised” (High, 
Medium or Low) 
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Assessing Functional Test & Product 
Diagnosability 


Summary Fault Analysis Test Performance Analysis 
1. Overall, how likely are test 1. How likely are specific failures 1. How good is a specific test at 
escapes? to escape? catching escapes? 
2. Overall, how difficult is it to 2. How difficult is it to 2. How much does a test 
troubleshoot this product? troubleshoot specific failures? contribute to diag resolution? 
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Automating Diagnostics 


Fault Detective 
Model 
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Diagnosis 


Test Results 
Fault Detective Diagnostics Engine 
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Summary of Benefits/Results 


Near 100 percent diagnostic accuracy 


Incremental effort is minimal (for A/D, coverage requirement is mandatory on most 
programs)... “it’s a no-brainer” 


Manufacturing/Production 


Perform self-diagnosis after final assembly, before final verification. 


1. Standardized test documentation > Consistency across the company 

2. Coverage assessment > Fewer test escapes 

3. Diagnostics assessment > Predictable time-to-volume 

4. Automated diagnostics > Fewer resource requirements (fewer repair technicians & less capital equipment) 


> IP protection 


Service Centers/Depots 


Perform self-diagnosis before manual troubleshooting. 


1. Automated diagnostics > Fewer resource requirements (less time ramping-up technicians at each service depot) 
> Faster turn-around time 
2. Coverage assessment > Fewer no-trouble-founds 


In the Field 


Peform diagnostic from anywhere in the world. 


1. Automated diagnostics > Fewer resource requirements 
> Increased product up-time 
2. Coverage assessment > Fewer failures in the field 
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AFOTEC’s Perspective 
on 
Operational Suitability 
Testing 


Mr Jerry Kitchen 
AFOTECICA 


Approved for public release, Presentation date: 13 Mar 2007 
distribution unlimited Release date: 4 Mar 2007 


es Overview 


° Myth Busters 

° What’s Changed? 

° Suitability - An Afterthought? 
® Operational Test Construct 

° Acquisition Efficiencies 


es Myth Busters 


° New rating scheme — no more E&S 
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° New rating scheme — no more E&S 
° E only - it’s all about effectiveness 
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° New rating scheme — no more E&S 

° E only - it’s all about effectiveness 

° ‘ilities — things of the past 

°® Stoplight chart — no objective analysis 


acy Myth Busters 


° New rating scheme — no more E&S 
° E only - it’s all about effectiveness 
° ‘ilities — things of the past 


°® Stoplight chart — no ene 


es What’s Changed? 


° Customer-focused reports 
— Accurate, balanced, complete, clear, concise, cogent, and 
timely 
— Warfighter community 


=» Can a unit equipped with the weapon system accomplish 
the mission? 


— Acquisition community 
as Is the weapon system effective and suitable? 
® Capability-based OT&E 
— No longer “Pass / Fail” 
» Effective / suitable with limitations impacting operations 
— Mission Capability Rating 


MC RATING DEFINITION 


FULLY MISSION —— 
CAPABLE Mission can be accomplished in the intended operational 


environment 
(FMC) 


MISSION — — . 
CAPABLE © Mission can be accomplished, but with increased operational 


cost 
(MC) 


PARTIALLY MISSION | Only some aspects of mission can be accomplished 
CAPABLE And/Or 


(PMC) (Y) Mission can only be accomplished under some conditions 


NOT MISSION 
CAPABLE @ Mission cannot be satisfactorily accomplished 
(NMC) 


NOT RESOLVED C) Insufficient information to support a resolution 


es Suitability — An Afterthought? 


° U.S. Code Title 10 requires AFOTEC to determine BOTH 
operational effectiveness and operational suitability 

° Operational effectiveness is the overall degree of 
mission accomplishment of a system ... (DAG 9.4.4) 

° Operational suitability is the degree to which a system 
can be satisfactorily placed in field use, with 
consideration given to reliability, ... (DAG 9.4.5) 

° AFOTEC fulfills this mandate through the Operational 
Test Construct 


Operational Test Construct 


® Methodical, repeatable test construct is crucial for 
accurate, balanced, and complete reports 


— Foundation for all follow-on planning, evaluation, and 
reporting activities 


— Must capture CONOPS, desired capabilities, and defined 
requirements found in capability documents 


° Extremely challenging test team activity 


— Involves participants across acquisition community 
=» Each brings own perspectives, priorities, and agendas 


° Test Constructs must use clearly defined terminology 


‘eg j.§ DAG “OT&E Best Practices” 


Focus on the mission(s) that will be accomplished 
Identify the operational capabilities 

Develop measures and data requirements 

Assess the degree of mission accomplishment 


Link mission accomplishment to the key operational 
capabilities 


Defense Acquisition Guidebook, Ver 1.6, Chapter 9.8.2, 07/24/2006 43 
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AFOTEC Test Construct 


Mission Statement 


Operational 
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Evaluation Taxonomy 


Test Measures Critical Operational Issues Mission Capability Resolution 


eee DR Ops Experience 
. & Judgment 


++ DR Ops Experience 


Test DRs Ops Experience 
+ S  & Judgment 


Events & Judgment 


on COI? 


Vie. : 


. Significance? 


Severe / Degree of 
Impacts? 


Substantial 
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AFOTEC Construct Is not 
designed to facilitate 
meM math 


~ ’ . 
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es Delivering Capability 


® Capability stool is supported by 
three legs 
— Cost 
— Schedule 
— Performance 
® Current acquisition philosophies 
make for a very shaky stool 
— Promise the moon 
— Technology “creep” and “reach” 
— Get it now, fix it later 
— Fear of failure 
— Rush to failure 
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The Promise 


Give the warfighter the tools to get the job done 


\ 
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Fail to Deliver 


Fail to design, build, field, and sustain capable 
/ weapon systems 


acy Truth in Advertising 


Acquire weapon systems in attainable, capable increments 


CAPABILITY 


a ox ‘ 


Warfighter Focused! 
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™ Joint Mission Environment Test — 
Capability (JMETC) 


23 Annual National Test and 
Evaluation Conference 


Mr. Richard Lockhart 
Principal Deputy Director 
Test Resource Management Center (TRMC) 


March 13, 2007 


TRMC Functions 


Sec. 231, FY 2003 National Defense Authorization Act 
DoD Directive 5105.71, March 8, 2004 


¢ DoD Field Activity 
¢ Direct Report to USD(AT&L) 


wroew SES Director 


Annual T&E Budget 
Certification 
(Military Departments 
and DoD Agencies) 


Biennial 10-Year 
Strategic Planning 


Policy 
Oversight 


Administer 
T&E Investment 


Programs 
(CTEIP, T&E/S8&T, 
JMETC) 


Oversee T&E Budgets 
and Infrastructure 

(MRTFB and 
other test facilities 


T&E 
Workforce 


TRMC Vision 


“The Department of Defense T&E Ranges and Facilities 
will be fully capable of supporting the Department with 
quality products and services in a responsive and 
affordable manner.” 


Weyer ig stayfelrels or tals 
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TRMC Organization 


r. John B. Foulke 


Principal Deputy — 
Richard L. Lockhart. 


A, 


/ 


Front Office Group 
Chief Financial Officer, Sr Military Asst, — 
Program Support, Executive Support) 
Test Resources Joint Investment Programs & Policy Test Infrastructure Strategic Planning 
Gary Carter Richard L. Lockhart Fred Myers Carlos Tirres (Acting) 
Policy CTEIP 
Phyllis Ferguson Derrick Hinton ali 


George Rumford 


Testing in a Joint Environment 
Background 


¢ March 2004 — SPG: “Joint Testing in Force Transformation” 


— Policy — Developing and fielding joint force capabilities requires adequate, 
realistic test and evaluation in a joint operational context 


— Direction — DoD will provide new testing capabilities and institutionalize the 
evaluation of joint system effectiveness 


— Action — DOT&E lead development of a Roadmap to define changes to 
ensure that T&E is conducted in a joint environment and facilitates the 
fielding of joint capabilities 


¢ November 2004 — DEPSECDEF approved Roadmap, validated SPG 


¢ December 2005 — PBO7 PDM approved Change Proposal 


— Joint Mission Environment Test Capability (JMETC) Program Element 
under USD(AT&L)/ TRMC for execution 


¢ FY 2006 — Conducted five capability demonstrations of JMETC 


¢ October 2006 — Establishment of JMETC Program Management 
Office in Crystal City, VA 


¢ Problem 
— No corporate approach 


— Multiple unique solutions — AF-ICE, Navy DEP, JNTC 

— Multiple networks — DREN, DISN-LES, DATUMS 

— Multiple standards — TENA, HLA, DIS 

— Unique program-specific solutions for distributed LVC events 


¢ Impact 
— Difficult, time-consuming to conduct distributed LVC events 
— Time consuming to establish security agreements 
— Costly duplication of infrastructure 
— Difficult to conduct joint testing 
— Difficult to integrate Test and Training 


What is JMETC? 


¢ Acorporate approach for linking distributed facilities 
— Enables customers to efficiently evaluate their 
warfighting capabilities in a joint context 
— Provides compatibility between test and training 


¢ Acore, reusable, and easily reconfigurable foundation 


— Consists of the following products: 
¢ Persistent connectivity 
¢ Middleware 
¢ Standard interface definitions and software algorithms 
¢ Distributed test support tools 
¢ Data management solution 
¢ Reuse repository 


— Provides customer support services 


JMETC Responsibility 


Product Implementation and Sustainment 


° Persistent Connectivity: Establish and maintain dedicated VPN on 
SDREN with standing security agreements. 


¢ Middleware: Adopt and sustain downloadable TENA universal data 
exchange software. 


¢ Standard Interface Definitions & Software Algorithms: Adopt and 
sustain downloadable TENA-established interface definitions and software 
algorithms (e.g., Radar, TSPI, coordinate and unit conversions, etc.). 


¢ Distributed Test Support Tools: Collaborate with CTEIP and the 
Services to adopt and sustain InterTEC software tools and other existing 
software tool capabilities that meet JMETC customer needs. 


¢ Data Management Solutions: Collaborate with CTEIP to develop new 
capabilities that meet JMETC customer needs. 


¢ Reuse Repository: Establish and maintain on-line web portal with 
relevant distributed event information (latest middleware, software 
components, documentation, lessons learned, etc.). 


Implementation dependent on funding and customer need ; 


Test & Training Enabling Architecture 
(TENA) 


— Developed, upgraded, and sustained by CTEIP and JNTC 

— Middleware that provides a single, universal data exchange solution 
— Common for test and for training (core standard in JMETC and JNTC) 
— Available for download at www.tena-sda.org for free 


¢ TENA provides: 
— Interoperability among range systems, hardware-in-the-loop 
laboratories, and simulations in a quick, cost-efficient manner 
— A capability to rapidly and reliably develop LVC integrations 
— A set of community-agreed object models that define the data elements 
used in LVC integrations — maximizes reuse from event to event 
— An auto-code generator to drastically reduce TENA incorporation time 


¢ Next version of TENA (version 6.0) will: 
— Provide advanced data filtering (only data of interest sent over the wire) 
— Improve fault tolerance and embedded diagnostics 
— Be released in Fall 2007 


JMETC Infrastructure 
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Joint Operational Scenarios 


Systems 
Under ~ 
nee a Ge sey 
Integrated Virtual Hardware Installed Environment Threat 
Test Prototype in the Sistem Generator Systems 


| ee) e) Test 
Resources a Facility 


TENA 
TENA TENA TENA TENA . TENA 
Common Common Toxeynitiiteyal ommon Common 


Common ; 5 
Middleware Middleware Middleware Middleware Middleware 


Middleware 
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Reuse 
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Distributed Test Data Management 
Support Tools Solutions 


¢ InterTEC Spiral 1+ 
— Conducted Air Combat Mission Thread using Live (8 F-16, 2 F-22, 1 E-2C), 
Virtual (F-35, F/A-18, F-15, CVN), and Constructive (threats systems) 
— Linked 8 sites: Edwards, Eglin, Pt. Loma, Pt. Mugu, China Lake, Pax River, JITC, Fort Worth 
— Capability accredited by JITC as suitable for interoperability certification of F-35 


¢ Army 3CE 
— Integrated M&S resources among 3 Army Commands (TRADOC, RDECOM, ATEC) 
— Linked different data exchange solutions (TENA and HLA) via gateway 
— Linked 6 sites: White Sands, Redstone, Aberdeen, Ft. Sill, Ft. Belvoir, Ft. Kno OX | 


¢ Information Operations (lO) Range 


— Distributed multiple video streams of IO effects from remote sites 
to JFCOM during the Austere Challenge ’06 exercise 


— Linked 10 sites, and redistributed video to 35+ observation sites 


- Test & Training collaboration (Technical Alignment with INTC) me 


— Demonstrated integration commonality with JNTC instrumentation used during Weapons & 
Tactics Instructor and Red Flag Alaska training exercises (as well as Talisman Sabre) 


— Linked 3 sites: Yuma, Quantico, Suffolk, plus two Alaskan ranges 
— Developed a TENA interface to GCCS (in <1 month) to display range data on GCCS 


Demonstrated the JMETC concept Is viable and able to support distributed tests 
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JMETC Responsibility 
Customer Support 


JMETC supports customer needs at customer request 


The JMETC Team: 

¢ Assists in understanding JMETC and how to use it for testing 

¢ Assists with T&E strategy and requirements 

¢ Supports limited event planning, preparation, and execution 

¢ Reviews and certifies JMETC products for corporate use 

¢ Integrates new nodes onto JMETC VPN with security agreements 

¢ Augments DREN with sites critical for joint testing (maximizing reuse) 
¢ Measures JMETC infrastructure performance 

¢ Provides Help Desk to assist JMETC product users 

¢ Provides semi-annual TENA training classes 


Prioritization of effort is based on funding available 
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¢ The customer is responsible for: 
— Defining requirements 


— Providing test assets and resources 


— Requesting and funding field assistance: 
o Technical integration support, including site verification 
o JMETC product training 
o Detailed event planning, preparation, and execution 

— VPN usage fees for connectivity and bandwidth 


— Unique middleware, object model, and software tool 
development and upgrades 


¢ Sites not on JMETC VPN build plan may fund their own 
addition to JMETC infrastructure 


¢ Two ways to become a part of the JMETC infrastructure: 


— Being on government contract to support a program or test event 
using JMETC 


— Contact the JMETC Program Office 


¢ TENA middleware and object models freely available — 
over 27 companies members of TENA Architecture 
Management Team 


— Technical forum providing open dialogue between users and 
TENA developers 


— Used to identify issues, vet concerns, debate solutions, and 
agree on way forward 


Industry is a key component 


in a successful DoD “corporate approach” 
to linking distributed facilities 


JMETC Leadership & Governance 


JMETC 
Chain of Command 
Hon. Kenneth J. Krieg Membership 
EEDIAreE) - JS J8 Co-Chair 


* OSD TRMC Co-Chair 
* DOT&E 

¢ JFCOM 

° JS J6 

¢ Army 

¢ Navy 


JMETC 
Governance 


Dr. John B. Foulkes 
Director, Test Resource 
Management Center (TRMC) 


Richard L. Lockhart 
Principle Deputy, TRMC 
Deputy Director, JIPP 


JMETC 
Senior Leadership 
Group (SLG) 


¢ Air Force 


- USMC 
* (OUSD) P&R 


Chip Ferguson JMETC * (OASD) NII 
JMETC Program Manager Advisory Group * (OSD) PA&E 
* (OUSD) | 
* DISA 


George Rumford JMETC ; 
Systems Engineering Lead a (OUSD) AT&L 
(Acting) 
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JMETC Support Groups 


¢ JMETC Advisory Group 
(GS-15/O-6-level headquarters representatives of T&E community) 


— Focuses is on overall program planning, direction, and priorities 
— Assists in identifying customers and priorities 
— Facilitates collaboration with Service initiatives 


¢ JMETC Users Group: 
(Acquisition program offices, technical experts, and ranges that are 
potential users of JMETC infrastructure and products) 


— Focuses is on technical requirements and solutions 

— Makes recommendations to improve JMETC processes and 
procedures 

— Advises T&E community on TENA priorities prior to AMT meetings 
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JMETC FY07 Activities 


¢ Stand up the Program Management Office 
— Fill positions 
— Develop Users' Guide and Long-Range Plan 


¢ Stand up JMETC Users Group 

¢ Establish VPN based on customer requirements 

¢ Support JCAS 07-03 event -— JECOM/JTEM/AF-ICE 
¢ Support InterTEC Spiral 2-Build 1 


¢ Support planning for FY08 customers and events 
InterTEC (Spiral 2-Build 2) 

— CVN-21 

— FCS 

— SIAP (JCHE-5) 


JMETC Benefits 


¢ Provides Department-wide capability for: 
— T&E of a weapon system in a joint context 


— DT, OT, Interoperability Certification, Net-Ready KPP compliance 
testing, Joint Mission Capability Portfolio testing, etc. 


¢ Provides test capability aligned with JNTC 
— Both use TENA architecture 
— Enables joint test and training 


¢ Reduces time and cost by providing 
— Readily available, persistent connectivity 
— Standing network security agreements 
— Object model auto-code generation tools 
— Tools that allow easy access to previous solutions and lessons learned 
— Corporate data exchange middleware 


¢ Provides Customer Support 


— For JMETC products 
— For LVC distributed test expertise 
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JMETC Points of Contact 


Program Manager: Chip Ferguson 
703-604-0350 x138 
chip.ferguson@osd.mil 
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SY or © 


Systems Engineer: George Rumford 
703-601-5233 
george.rumford@osd.mil 


Operations Planning: Len Zimmermann 
703-604-0350 x141 
leonard.zimmermann.ctr@osd.mil 


Long Range Planning: Cynthia Lindberg-Ross 
703-604-0350 x146 
cynthia.lindberg-ross.ctr@osd.mil 


Back-Up Slides 
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Industry Involvement 


¢ TENA solution is free 
— Over 27 companies already members of TENA Architecture 
Management Team 


¢ Industry participation includes: 


*AMTEC eJacobs Engineering «Scientific Research 

¢eAnteon Corporation «Johns Hopkins Corporation (SRC) 

*BAE Systems University «Scientific Solutions, 

¢BMH Associates ¢Lockheed Martin Inc. (SSI) 

¢Boeing ¢MAK Technologies ¢SRI, International 

¢Computer Sciences ¢NetAcquire ¢ Trideum Corporation 
Corporation (CSC) «Northrop Grumman «Weibel 

«Cubic Defense ¢eRaytheon ¢Wyle Laboratories 


*DRS Technologies °Samoff Cooporation «Virtual Technologies 

¢Electronic Warfare ¢Science Applications Corporation 
Associates (EWA) International 

¢eEmbedded Planet Corporation (SAIC) 

EMC 
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ARMOR HOLDINGS 


AEROSPACE & DEFENSE GROUP 


ADVANCING SURVIVABILITY 


Testing in the Design Phase 


Regis Luther 


Measuring and Validating Reliability 
Modeling/Simulation/Validation 


Initial Final 
RAM-D RAM-D 


Analyze 
Requirements 
and Results 


Analysis 4 Simulation WEViTer hated ay 


Warranty 


AER l ENSE GROUF 


FROSPACE & 


ADVANCING SURVIVABILITY 


Copyright © 2006 by Armor Holdings, Inc. 


Model Based Definitions 
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| 
S&S Blast Mine Test 
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a 
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ee 4 ARKMC we a! | 


=» Founded in 1996 - Headquarters in Jacksonville, Florida 


a» Public Company (NYSE - AH) 
Market Cap of $1.898 billion (as of 8/16/06) 


» Revenue 
- 2004 Revenue - $980 million 
- 2005 Revenue - $1.637 billion 
- 2006 Revenue - $2.4 billion anticipated 


» 6,200 Employees in Three Business Units 
2005 Revenue 


Aerospace & Defense Group (ADG) $1,189m 
+ Stewart & Stevenson, Pinzgauer $ 726m 
Products Group $ 309m 
Centigon $ 139m 


» Manufacturing Operations 


- AL, AZ, CA, FL, KY, MA, MI, NH, OH, PA, TN, TX, WY [>> 
- Brazil, Colombia, France, Germany, Mexico, United 
Kingdom, Venezuela, Switzerland ARMOR HOLDINGS 


ADVANCING SURVIVABILITY 


A 
AH 2005 Revenue* 


2000 RSE 
Group 19% nm 
” q Geniigae 
erospace 8% 
1500 Defense Group 
73% 
1000 
500 
0 
1997 1999 2001 2003 2005 


* Exclusive of 2005 Stewart & Stevenson and Pinzgauer revenue of $726 million 
AEROSPACE & DEFENSE GROUP 
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AH Global Reach 


. Centigon 
Pinzgauer Bremen, Germany 
Fareham & Guildford, UK / 

\ 


API 
Manchester, UK 


wae 


16 facilities* 
United States 


Centigon 
Lamballe, France—— 


Corporate 
A Jacksonville, FL 
Safariland, ABA yy, 
Tijuana, Mexico ee a 


Tactical Vehicle Systems / 
Sealy, TX j 


—~__ Centigon 


: Projectina 
Caracas, Venezuela 


Herbrugg, Switzerla 


/ 


/ % ‘s 
Centigon ’ 7 
D.F., Mexico 
‘ 
\ 
: Centigon 
Centigon Sao Paulo, Brazil 


Bogota, Colombia 


Tactical Vehicle Systems ¥ V4 


Adelaide, Australia 


* Aerospace & Defense Group _ 

* Centigon (Mobile Security) ~\ 

* Products Group ey 
Natio) me ale) cel lare fmar- eye) (6 


. . AEROSPACE & DEFENSE GROUP 
relcete[Ulemcom-li merelUlaitai-t-m lame) [6[- : ; 
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Test Capabilities 


= Rolling road and test track facilities 
= Complete vehicle testing 

m Military Packaging 

= Ballistics testing 

= Materials testing 

m Static testing 

m Dynamic testing 

m= Full-scale impact testing 

= Outdoor and indoor drop towers 

= Horizontal sled 


m High speed photography and video 


US Military - FAA - Automotive - Commercial 


In-house test capabilities for internal R&D; certified outside labs validate or qualify systems 
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Test Track 


Random 3.3 rms 
Offroad profile 


V 


Fence Line 


West Loop 
90’ diameter 
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Random 2.4 rms 
Offroad profile 


Frame Twister 
15% and 20% 
opposing grades 


8”-10” Bumps (1 to 4) 
4’ to 33’ spacing 

10” Pot-Holes 
7’,10',13’ spacing 


Random Washboard 
2” to 5” bumps 
5' to 25’ spacing 

2.0 rms 


2” Washboard 
uniform 
2’ spacing 


Visitor Path \ Existing Offroad Random Radial 
6'-0” wide Rails = 0.3 rms Washboard Loop 
Moguls = 3.0 rms heli 
18” Step and sas or ole 
24” Step Vee Ditch 08 “DUMPS 


20 degrees both sides 
Approach Ramp 
20 degrees one side 


5’ to 20’ spacing 


>" 


ARMOR 
& DEFENSE GR¢ 
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Existing brake 
test area 
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Ultra-Reliability 


Demonstrated Reliability 


FMTV AL TECOM Test #1-VG-120-MTV-037, 5 vehicles, 120k total miles, S&S FMTV AL 


TECOM Test #1-VG-120-MTV-019, 28:vehicles, 488K total miles 


FMTV AO 
M1070 HET TECOM Test #1-VS-120-HET-003, 3 vehicles, 57k total miles, M1070 HET 
M44A3 TECOM Test #1-VG-120-035-044/050, 2:vehicles, 10k total miles, M443 

HMMWV |< 7<6/)) TECOM Test #1-VG-120-HMV-079/085, 9 vehicles, 180k total miles, HMMWV mm Demonstrated MMBHMF 
0 2,000 4,000 6,000 8,000 10,000 12,000 14,000 16,000 18,000 
TECOM = Test and Evaluation Command MMBHMF = Mean Miles Between Hardware Mission Failure 

* Fleet average of test vehicles including all failures attributed to CR changes and A1 baseline. &D 

AEROSPACE & DEFENSE GROUP 


ADVANCING SURVIVABILITY 
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Raytheon 


Customer Success Is Our Mission 


Statistically-Based 
Test Optimization Strategies 


23 Annual NDIA Test & Evaluation Conference 
March 13, 2007 


Neal Mackertich, Ph.D 
Raytheon Integrated Defense Systems 


© 2007, Raytheon Co., All Rights Reserved 


Raytheon 
Best Practices in Statistically-Based Test Optimization 


Why, What , When & How: There are lots of meaningful 
testing questions and opportunities to explore... 


The focus of this presentation is to share with you three 
industry cited best practices around the use of 
Statistically - based test optimization strategies: 


e Usage-based Statistical Testing 


e Combinatorial Design Methods 
e Critical Parameter Management 
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N. Mackertich 


Raytheon 
What to Test? 


Okay, let’s say we need to test a 
system or subsystem... 


How do we typically determine what 
test cases to run? 
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Raytheon 
Usage-Based Statistical Testing 


e Use-Based Statistical testing emphasizes the operational 
scenarios most likely to occur 


The Universe of possible test cases 


Least Likely Most Likely 
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Raytheon 
Usage-Based Statistical Testing 


e Testing based on the way the system / product is to be 
operational used. 


e A “Use Case” state diagram (in the form of a Markov Model) 
is used to generate a representative, random sample usage 
So Statistical methods can be applied to model System 
behavior. 


e Specific test case generation is accomplished using input 
stimuli selected via a random walk through the Markov 
chain. 
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Raytheon 
Sample Usage Model 


¢ States - represent pertinent usage 
history i.e.. The state of the software 

Q, from an external user’s perspective. 
Se 


¢ Arcs - represent state transitions caused 
by applying stimuli 


* Transition probabilities - simulate 
expected user behavior 


N. Mackertich 
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Raytheon 
Usage-Based Statistical Testing- Case Study Results 


e Escaping defects were less than previous methods 
(1.16 defects per KSLOCs Vs. 6-defects/ KSLOCs) from 
previous testing method. 


e Development costs met budgets. 
e Major functions were integrated quickly. 


e Poll was taken with the leads about whether to use this 
Statistical testing on a follow-on program; results were 
unanimous to use statistical testing for next project which 
has already started!! 
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Raytheon 
What are Combinatorial Design Methods? 


e A testing methodology in which a subset of all possible 
combinations is chosen such that all N-way combinations 
are tested. 


¢ Covering all 2 way combinations would require that for any 
two factors A and B, where Ai and Bi are valid levels for A 
and B, there is a test for all Al and BI combinations. 
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N. Mackertich 


Raytheon 
Why not DOE? 


The use of proven statistical / combinatorial 
methods can be very helpful here... 


But why not just use Design of Experiments 
(DOE) techniques? 
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Raytheon 
CDM Advantages 


e The use of N-way combinations provide reasonable, balanced 
coverage across the test space. 


e More realistic than full/fractional designs 
— Compatible with constraints 
— Compatible with factors at different levels 
— Can account for previous test 


e Drastically reduces the total number of test cases when compared to 
all combinations. 


e Since generating test cases is very quick and simple, there are no 
major barriers to using CDM as part of the testing process. 


e Can be used in almost all phases of testing. 
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Raytheon 
Black-Box Testing Scenario 


Black-box type testing geared to functional requirements of an application. 


Example Application Graphics manipulation function that converts from one format to another. 


Inputs Source Format (GIF, JPG, TIFF, PNG) 
Dest. Format (GIF, JPG, TIFF, PNG) 
Size (Small, Med, Large) 
# colors (4 bit, 8 bit, 16 bit, 24 bit) 
Destination (Local drive, network drive) 
Windows Version (95, 98, NT, 2000, Me) 


Outputs Correct conversion (True or False) 


lf Destination Format is GIF, then # colors cannot be 16 bit or 24 bit. 
All Combinations 1920 Test Cases 


2 Way Combinations 22 Test Cases 
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Raytheon 
Regression Testing Scenario 


Definition Re-testing after fixes or modifications of the software or the environment in which the 
software operates. 


Example Application Graphics functionality on previous page must be expanded to support a new file format 
(WMF) and a new OS (Windows XP). 


Inputs Source Format (GIF, JPG, TIFF, PNG, WMF) 
Dest. Format (GIF, JPG, TIFF, PNG, WMF) 
Size (Small, Med, Large) 
# colors (4 bit, 8 bit, 16 bit, 24 bit) 
Destination (Local drive, network drive) 
Windows Version (95, 98, NT, 2000, Me, XP) 


Outputs Correct conversion (True or False) 
If Destination Format is GIF, then # colors cannot be 16 bit or 24 bit 


All Combinations 1620 Test Cases 
2 Way Combinations 32 Test Cases 
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Raytheon 
CDM- Case Study Application 


e Testing program requirements 
Factory Acceptance Test (FAT) Dry Run 
— FAT 

Site Acceptance Test (SAT) Dry Run 

— SAT 


e Not realistic to do exhaustive testing of all 144 possible 
System test scenarios 


e Quasi-Exhaustive strategy invented 

100% of tests for FAT Dry Run 

— 10% of tests, selected at random, for FAT 

— 50% of tests, selected at random, for SAT Dry Run 
10% of tests, selected at random, for SAT 
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Raytheon 


CDM Case Study- A Comparison of Strategies 


Quasi-Exhaustive Strategy | FAT Dry Run FAT SAT Dry Run SAT 


Number of test cases 


144 


15 


72 


alts: 


Time spent writing test 
procedures 


49.00 hours 


0.00 hours 


8.00 hours 


0.00 hours 


Time spent performing data 
paths test procedures 


73.25 hours 


8.75 hours 


55.00 hours 


12.25 hours 


Number of persons involved 
in testing 


3 


3 


4 


Total Labor Hours 370.5 hours (46.3 days) 


CDM Strategy FAT Dry Run FAT SAT Dry Run SAT 


Number of test cases 


12 


12 


12 


12 


Time spent writing test 
procedures 


16.00 hours 


0.00 hours 


3.00 hours 


0.00 hours 


Time spent performing data 
paths test procedures 


7.25 hours 


7.25 hours 


10.00 hours 


10.00 hours 


Number of persons involved 
in testing 


3 


3 


4 


Total Labor Hours 118.0 hours (14.8 days) 


N. Mackertich 
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Bottom Line Comparison Raytheon 
(Time-to-Market Savings) 


e CDM Strategy was superior to Quasi-Exhaustive Strategy 
— Schedule savings = 68% 
— Cost savings (labor) = 67% 
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Raytheon 
What is Critical Parameter Management? 


e A disciplined methodology for managing, analyzing, 
and reporting technical product performance. 


e A process for mathematically linking system 
parameters for sensitivity analysis and optimization of 
critical performance threads. 


e A strategic tool for improving product development by 
unifying and integrating systems, design, 
manufacturing and test activities. 


CPM = TPMs + Other parameters critical to functionality, cost, schedule or customer 
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Raytheon 
Performance Analysis 


“The System Can....” 


SaEEs X’s 
Opportunity 


CPM Statistically Flow-Up Design, Supplier and Manufacturing 


Capabilities Exposing Performance Risks and Opportunities 
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Raytheon 
Whitepaper Management and Task Delegation 


Engineering Documents and 
Whitepaper Analyses 


Attach Engineering Documents, Models, and Whitepaper Analysis. 


Connect people to analyses, requirements, and performance measures. 
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Raytheon 
CPM Program Benefits 


e Facilitate Analysis 


— Statistical modeling & optimization of the performance — cost trade space 
— Real-time System-level sensitivity analysis 
— Connects analyses between system, subsystem and component levels 


e Improve Collaboration 
Shares technical analysis and knowledge 
Links ownership to parameters 


Mathematically connects Program teams and parameters to understand 
requirement flow-down 


Captures and leverages invested intellectual capital for future business reuse 


e Enable TPM Management and Reporting 
— TPM design margins are statistically tracked over product lifecycle 
— Automated, real-time TPM data gathering / report generation 
— Reconciliation of requirement allocation and engineering design capability 
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Raytheon 
DFSS Statistical Performance Analysis 


Design Variables 


Response 


Y = f (A, B, C, D, E, F,...,M) 


Y 


° 
8 


Allocation/Flow Down 
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Raytheon 
Statistical Performance Analysis Results In... 


Parameters : oe PDF(Y) 
‘ Function ¢ A prediction of the 
F(A, B, C,D,...) Response Statistical 2° CU 
Properties z 
B en ee ee ae ee 
¢ A prediction of the Prob(LL<Y<UL) 


Response Probability of Non- 


C y Compliance 2. 3 
E 4 6d 


e An assessment of the grees seme ae 


4 
D Contribution of Parameter -<<—<——se : 
“ . A-Pavg 35 EEE 
Variation to Response ee a 
Performance and Cost ccm |) 
E cnt ae : ie | 
cap ae = 
I-Noise Figure | -.12 gE 
L-Rep. Freq. -.03 i : 


Results from Crystal Ball® Monte Carlo SW 
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Raytheon 
Traditional TPM Stoplight Reporting 


CDR 


TPM Number Description Aug '05 Sep '05 Oct'05 Nov'05 Dec '05 Jan '06 


e Meets Requirement with Margin 
Meets Requirement with No Margin 
R Does Not Meet Requirement 


QANDADADAAADNDAAD 


Previous TPM Reporting / Tracking method is vague and ambiguous 
with respect to the design margin for each metric. TPM report was only 
tracked and managed on a monthly basis for the tracking book. 
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Raytheon 
Statistically-Based TPM Reporting / Management 


Statistically track design capability and 
requirement by establishing upper and lower 
limits 
Monitor design capability and requirement 
convergence over product lifecycle: 

- IPDR, PDR, CDR etc. 


-Design, Manufacturing, Integration, Test 


Probability of Non-Compliance 


Oct-15 12:00 Nov-14 12:00 Dee-14 12:00 Jan-13 12:00 Feb-12 12:00 


DY Target Variance ~—@= Target Nominal Design Variance = Design Nominal 


Cp and Cpk 


Probability of 
LSL defect. 


Probability of 
USL defect. 
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Raytheon 
New Automated TPM Reporting Format 


TPM design margins are statistically tracked real-time 


Block Name: TPMs BOM ID#: 2/9/2006 
Target Value Design Model Prediction Validation Test Data 
CFR (Y) Spec Owner Units Lower Upper 
Code Limit |Nominal] Limit {Nominal} Mean | Std. Dev. Mean | Std. Dev.}| Cp |Cpk 

TPM 1 0.9400 | 0.9600 | 0.9990 | 0.9672 | 1.9040 | 0.6238 | 0.0158 | 1.9836] 0. 

6.0000 | 8.0000 | 11.0000} 7.0000 | 9.5000 | 0.5833 0.3571 | 0. 
TPM 3 43.0000] 45.0000} 49.0000] 43.0000] 42.7500] 0.3750 | 2.6667 |-0.72 ' 

14.0000 17.0000} 19.0000] 16.0000] 16.5000} 0.4167 | 2.0000 | 1.5000 |0. 


TPM 4 


TPM 2 R 


TM 5 
TPM 6 


0,6000-[0.7000]|0.0383 | 4.0000 | 2.8883]p,0000| | |] 


0.4500 | 0.5000 | 0.4500 | 0.4550 | 0.0042 | 4.0000 3.4333/).0000; | sd 


0.1000 | 0.1200 | 0.1600 | 0.1250 | 0.1250] 0.0008 |12.0000] 9.5000 #S. 
4.0000 } 6.8000 | 8.0000 | 7.0000 | 7.0000} 0.0833 | 8.0000 | 3.5000 |). 


Meaningful stoplights 


based on statistical Compares requirement Includes sensitivity 
sensitivity allocation vs. current and statistical 
PRCT PCC) HPCE) design capability information 


Captures complete product lifecycle: 


requirements, design, manufacturing, testing, 
validation, etc. (other columns not shown) 
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Using Technology Readiness 
sessme nts AG Bo) to Assess the 


National Defense Industrial Association 
23d Annual National Test & Evaluation Conference 
March 12-15, 2007 


Dr. Jay Mandelbaum 


D4 D A Institute for Defense Analyses 
A 


850 Mark Center Drive « Alexandria, Virginia 22311-1882 


Outline 


¢ Life-cycle-related (LCR) Critical Technology 
Element (CTE) identification 

¢ Life-cycle-related Critical Technology Element 
assessment 


¢ References and resources 


What is a TRA? 


¢ Systematic, metrics-based 
process that assesses the 
maturity of Critical 
Technology Elements (CTEs) 
— Uses Technology Readiness 

Levels (TRLs) as the metric 

¢ Regulatory information 
requirement for al! 
acquisition programs 


# Not a risk assessment 
— Submitted to DUSD(S&T) for # Not a design review 


ACAT ID and IAM programs 


# Does not address system 
integration 


Critical Technology Element (CTE) Defined 


A technology element is “critical” if the system 
being acquired depends on this technology 


element to meet operational requirements with 
acceptable development cost and schedule and 


with acceptable production and operation costs 
and if the technology element or its application is 
either new or novel. 


Said another way, an element that is new or novel or 
being used in a new or novel way is critical if it is 
necessary to achieve the successful development 

of a system, its acquisition or its operational utility. 


CTEs may be hardware, software, manufacturing, or life cycle 
related at the subsystem or component level 


4 


TRL Overview 


¢ Measures technology maturity 
¢ Indicates what has been accomplished in the 
development of a technology 
— Theory, laboratory, field 


— Relevant environment, operational 
environment 


— Subscale, full scale 
— Breadboard, brassboard, prototype 
— Reduced performance, full 
performance 
¢ Does not indicate that the technology is right for 
the job or that application of the technology will 
result in successful development of the system 


Why is a TRA Important? (1 of 2) 


¢ The Milestone Decision Authority 
(MDA) uses the information to support 
a decision to initiate a program 
— Trying to apply immature technologies 
has led to technical, schedule, and cost 
problems during systems acquisition 
— TRA established as a control to ensure 
that critical technologies are mature, 
based on what has been accomplished 


¢ Congressional interest 


— MDA must certify to Congress that 
the technology in programs has 
been demonstrated in a relevant 
environment at program initiation 


— MDA must justify any waivers for 
national security to Congress 


Why is a TRA Important? (2 of 2) 


The PM uses the expertise of the assessment team and 
the rigor and discipline of the process to allow for: 
— Early, in depth review of the conceptual product baseline 
— Periodic in-depth reviews of maturation events documented 
as verification criteria in an associated CTE maturation plan 
— Highlighting (and in some cases discovering) critical 
technologies and other potential technology risk areas that 
require management attention (and possibly additional 
resources) 
The PM, PEO, and CAE use the results of the 
assessment to: 
— Optimize the acquisition strategy and thereby 
increase the probability of a successful outcome 
— Determine capabilities to be developed in the next 
increment 
— Focus technology investment 


PM responsibility 


Process Overview 


Set schedule 


=> Identify CTEs 


Collect; —. Coordinate CTEs 


data 


==> 


Assess CTEs; prepare TRA 


Coordinate and submit TRA 


OSD review 


PM responsibility 
Coordinate with S&T Exec 
Keep DUSD(S&T) informed 


PM responsibility 

Best Practice: Independent 
review team appointed by S&T 
Exec verifies 


PM responsibility 
Coordinate with S&T Exec 
Keep DUSD(S&T) informed 


S&T Exec responsibility 
Appoints independent review 
team to do it; PM funds it 


S&T Exec coordinates 
Acquisition Executive submits 


DUSD(S&T) responsibility 


Overview of Technology Considerations 
During Systems Acquisition 


User Needs & « Process entry at Milestones A, B, or C 


« Entrance criteria met before entering phase 


2 Evolutionary Acquisition or Single Step to Full 
Capability 


(Program 
Jo /e\ initiation) VX loc FOC 
Technology nN VW | )| Production & Operations & 
Development m i Deployment Support 


LRIP/OT&E © Decision 


eview 


Pre-Systems Acquisition Systems Acquisition Sustainment 


D0 —aiErEEEEE> CCD > CPD 


Outline 


¢ Introduction to Technology Readiness 
Assessments (TRAs) 


¢ Life-cycle-related Critical Technology Element 
assessment 


¢ References and resources 


What is a Life-Cycle-Related CTE 


¢ LCR technologies impact system 


supportability cost and/or time. They 
may: 


Reduce the logistics footprint 
Improve reliability/maintainability 


Lower operating, support, or maintenance Ie 
manpower requirements 


Enhance training 
Enhance human factors interactions 


Increase operational availability or 
readiness 


Improve the upgradability of the system 


Examples of Life-Cycle-Related CTEs 


Corrosion resistant material 

Thermal protection materials 
Supportable low-observable materials 
Obsolescence mitigation technologies 
Technical data automation technologies 
Material handling technologies 
Simulators or training simulations 


Autonomic logistics sensors, data links, or 
messaging transmission 


Advanced technologies that affect human factors 


Analysis technologies, such as automated 
diagnostics and prognostics 


Methods/algorithms for sensing or trend analysis 
Technologies that enable open systems architecture 


Why Be Concerned About Life-Cycle- 
Related CTES (1 of 2) 


¢ Definitional perspective 
— All costs encompassed in CTE definition 
¢ Policy perspective 


— DoDI 5000.2 states that “The project shall exit 
Technology Development when an affordable 
increment of militarily-useful capability has been 
identified ...” 


\ — CJCSI 3170.01E defines increment as “a militarily 


useful and supportable operational capability that 
can be effectively developed, produced or 
Ne acquired, deployed, and sustained...” 
— Mobility and logistics footprint are military 
capabilities and reliability and maintainability are 
military performance parameters 


Why Be Concerned About Life-Cycle- 
Related CTES 2 of 2) 


¢ Experiential perspective 
— Increasing operating and support cost is decreasing 
acquisition capability; therefore greater emphasis is 
being placed on life cycle related issues including the 
technologies that affect them 


— Operating and support 
costs will continue to 


Percentage of Systems Passing Operational Test 


increase, thereby making oy 
life cycle related 7 
technologies even more 50 efective 
Critical 20/4 =—= 
2071 | 
1077 | 


0— 
1984-1994 1995-1999 2000-2005 


57 systems 33systems 36 systems 


Source: 15 September 2005 David Duma presentation to the 


Enhancing the CTE Identification Process to Better 
Detect Life Cycle Related Technologies (1.3) 


¢ Modify determination of criticality 


— Does the technology directly impact an 
operational requirement? 


— Does the technology have a significant a4 
impact on an improved delivery ew’ e,% 
& R= 
schedule? a 


— Does the technology have a significant 
impact on the /ife cycle affordability of 
the system? 


Enhancing the CTE Identification Process to Better 
Detect Life Cycle Related Technologies ¢.:3) 


¢ Clarifying life cycle cost questions —- 
procurement cost component 


— How much does it cost to buy the 
component or subsystem with this 
technology? 


— Will the cost be significantly higher 
without the technology? 


Enhancing the CTE Identification Process to Better 


Detect Life Cycle Related Technologies ¢.:3) 


¢ Clarifying life cycle cost questions — O&S cost 


component 


Does the technology significantly reduce the 
logistics footprint? 

Does the technology significantly improve 
reliability/maintainability? 

Does the technology significantly lower operational, 
support, or maintenance manpower requirements? 
Does the technology significantly enhance training by 
some combination of lowering the resources needed or 
boosting its effectiveness? 

Does the technology significantly increase operational 
availability or readiness? 

Does the technology significantly improve the 
upgradability of the system? 


Best Practices for Life-Cycle-Related CTE 
Identification 


¢ Independent panel should 
evaluate “significance” as used >... 
in the questions for identifying 
CTEs 


¢ Include experts on appropriate 
LCR areas on the independent 
TRA panel 

¢ Once a CTE has been identified 
from a performance perspective, 
also evaluate if it is an LCR CTE 
as well 


— Must also apply “new or novel 
test 


An Example of a Technology Being Critical 
From Two Perspectives 


¢ The technology directly impacts an operational 


Active Electronic — The radar beam can be steered close to the speed 
Scanned Array of light, thereby enabling superior performance 
(AESA) Radar including air-to-air tracking at very long detection 

ranges, almost simultaneous air-to-air and air-to- 

surface mode capability, and enhanced situational 
awareness 


¢ The technology significantly impacts life cycle 
affordability 
— MTBCF ts greater than 15,000 hours for the array 
and greater than 1,250 hours for the entire system 


¢ Everything in the system is new 


Sources: June 28, 2005 Raytheon News Release, Raytheon’s Revolutionary APG-79 AESA Radar is Awarded a $580 Million Multi-Year Procurement Contract by 

the Boeing Company http://www.prnewswire.com/cgi-bin/micro_stories.pI? ACCT=149999&TICK=RTN&STORY=/www/story/06-28- 

2005/0003985043&EDATE=Jun+28,+2005; November 20, 2002 Raytheon News Release, Raytheon Demonstrates First Next-Generation AESA Capability at APG-79 

Event http:/www.raytheon.com/newsroom/briefs/112002.htm; Raytheon AN/APG-79 AESA data sheet 
http://www.raytheon.com/products/stellent/groups/sas/documents/legacy_site/cms01_050831.pdf. All source material copyright Raytheon Company - Rights 

reserved under copyright laws of the United States. Permission is granted by Raytheon Company for the U.S. Government to copy this material for evaluation 19 
purposes only. 


Outline 


¢ Introduction to Technology Readiness 
Assessments (TRAs) 

¢ Life-cycle-related (LCR) Critical Technology 
Element (CTE) identification 


¢ References and resources 
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Hardware and Manufacturing TRLSs 


Basic principles observed and reported 
Technology concept and/or application 
formulated 

Analytical and experimental critical a 
function and/or characteristic proof of 

concept 

Component and/or breadboard 
validation in a laboratory environment 
Component and/or breadboard 
validation in a relevant environment 
System/subsystem model or prototype 
demonstration in a relevant environment 
System prototype demonstration in an 
operational environment 

Actual system completed and qualified 
through test and demonstration 

Actual system proven through 
successful mission operations - 


Software TRLS 


. Basic principles observed and reported. 

. Technology concept and/or application formulated. 

. Analytical and experimental critical function and/or 

characteristic proof of concept e 

. Module and/or subsystem validation ina 50 

laboratory environment, i.e. software prototype 

development environment 

. Module and/or subsystem validation in a relevant 
environment G 

. Module and/or subsystem validation in a relevant 1 

end-to-end environment 

. System prototype demonstration in an operational 

high fidelity environment 

. Actual system completed and mission qualified 

through test and demonstration in an operational 

environment 

. Actual system proven through successful mission 

proven operational capabilities 
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Assessments Supported by Additional Information 


Example: TRL 6 Hardware Criteria 


Definition: System/subsystem model or prototype demonstration in 
a relevant environment. 


Description: Representative model or prototype system, which is 
well beyond that of TRL 5, is tested in a relevant environment. 
Represents a major step up in a technology’s demonstrated 
readiness. Examples include testing a prototype in a high-fidelity 
laboratory environment or in a simulated operational environment. 


Supporting Information: Results from laboratory 
testing of a prototype system that is near the 
desired configuration in terms of performance, 
weight, and volume. How did the test environment 
differ from the operational environment? Who 
performed the tests? How did the test compare 
with expectations? What problems, if any, were 
encountered? What are/were the plans, options, 
or actions to resolve problems before moving to 


the next level? os 


Modifying the CTE Assessment Process for 
Life-Cycle-Related Technologies (1 of 5) 


The definitions and descriptions corresponding 
to the various TRLS apply to LCR technologies 
Supporting information sufficient when: 
— Long-term effects do not have to be calculated 
¢ TRL 3 (analytical and experimental critical function 
and/or characteristic proof of concept) or lower 
— Long-term effects of the LCR CTE can be 
calculated analytically and the risk of error is 
minimal = 
¢ Shipboard weapons loader (SWL) provides a capability fora 
single operator to upload and download munitions while 
reducing operator workload and life-cycle cost 
¢ SWL demonstrations can confirm ship manpower reductions 
across the entire life cycle through a comparison with current 
crew requirements 
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Modifying the CTE Assessment Process for 
Life Cycle Related Technologies (2 of5) 


¢ Under certain circumstances, the supporting 
information should be augmented or tailored 
for the specific situation 
— When long-term effects cannot be accurately 
calculated analytically and the risk ofa 
miscalculation is large 
— When amplification of the Deskbook 
supporting information will be helpful in 
assigning the proper TRL 
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Modifying the CTE Assessment Process for 
Life Cycle Related Technologies (3 of 5) 


¢ Technologies that improve reliability/maintainability 
and correspondingly reduce the logistics footprint and 
operating, support, or maintenance manpower 
requirements 
— Additional supporting information should focus on the 
performance of the end item 
¢ The C-5 Reliability Enhancement and 
Re-Engining Program (RERP) is a 
comprehensive effort to improve | 
reliability, maintainability, and availability Hee 
¢ Long-term commercial experience used 


to assess TRL 7 for new propulsion 
system 
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Modifying the CTE Assessment Process for 
Life Cycle Related Technologies (4 of5) 


¢ Technologies used to protect against the 
environment (may also protect the 
environment) 


— Additional supporting information should 
focus on the performance of the material 
being tested 


¢ Using advanced high solid-edge retentive tank 
coatings instead of solvent-based paints; 
represervation of tanks represents the highest 
annual maintenance cost 


TRL 7 based on long-term commercial data on 
service life; reduced inspection, cleaning, 
preparation, and painting; and labor to apply 
the new coatings 
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Modifying the CTE Assessment Process for 
Life Cycle Related Technologies 6 of 5) 


¢ On-board and off-board technologies for 
analysis, status, or diagnosis of failure 


— Additional supporting information should 
focus on accuracy 


¢ The extensive use of predictive 
maintenance, conducted by networked on- 
board diagnostics and prognostics that 
pulse the system when issues arise (or are 
expected), is an important component of the 
Future Combat Systems (FCS) 


¢ When failures are random, physics-of-failure 
models do not exist; a statistical approach to 


prediction must be taken, but, data must be 
generated to support TRL greater than 4 


janned Systems, 


Pd <= 


neon Cartor | Senwcreenie 
on 


Eeuation (Manpackabte) UV 


Venice 
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Additional Life-Cycle-Related Supporting 
Information Has Been Developed 


Hardware TRL 6 Hardware TRL 6 Hardware TRL 6 
Definition Description Supporting Information 


System/subsystem Representative model or prototype Results from laboratory testing of a prototype 
model or prototype system, which is well beyond that of TRL system that is near the desired configuration 
demonstration in a 5, is tested in a relevant environment. in terms of performance, weight, and volume. 
relevant environment. Represents a major step up ina How did the test environment differ from the 
technology's demonstrated readiness. operational environment? Who performed the 
Examples include testing a prototype ina _ | tests? How did the test compare with 
high-fidelity laboratory environment or in expectations? What problems, if any, were 

a simulated operational environment. encountered? What are/were the plans, 
options, or actions to resolve problems before 
moving to the next level? 


Additional hardware supporting information 
for technologies to improve reliability/maintainability 


Analytical efforts at the subsystem level that estimate reliability (or reliability improvement if comparing to something 
already in existence). Efforts may encompass both an FMEA and failure-rate calculations for each of the subsystem 
failure mechanisms. Alternative corrective and/or preventive actions that could mitigate the most significant failure 
mechanisms should be identified. Maintainability analyses conducted to determine reliability-centered (failure-based) 
maintenance and condition-based maintenance strategies as well as a level of repair determination. Estimated 
support man-hours and spare parts’ needs meet expectations/requirements. Form-fit-function performance ensured. 


Additional hardware supporting information 
for technologies used to protect against the environment 


Material tested in a laboratory environment to provide assurance of its performance throughout its intended life cycle. 
Deliberately stressful/relevant environments are used to determine whether any degradation in performance occurs 
against known standards. Material interaction testing is conducted to ensure that no adverse chemical or other 
reactions occur in either the components being protected or other adjacent parts of the system. 


Software TRL 6 Software TRL 6 Software TRL 6 
Definition Description Supporting Information 


Module and/or sub- Level at which the engineering feasibility Results from laboratory testing of a prototype 
system validation in a of a software technology is demonstrated. | package that is near the desired configuration 
relevant end-to-end This level extends to laboratory prototype | in terms of performance, including physical, 
environment. implementations on full-scale realistic logical, data, and security interfaces. 
problems in which the software Comparisons between tested environment 
technology is partially integrated with and operational environment analytically 
existing hardware/software systems. understood. Analysis and test measurements 
quantifying contribution to system-wide 
requirements such as throughput, scalability, 
and reliability. Analysis of human-computer 
(user environment) begun. 


Additional software supporting information for analysis technologies 


Verify that faults can be detected/predicted using known faults in a simulated real environment such as a test cell or 
test platform not in use. Both Type | errors (actual faults not detected) and Type II errors (false positives) are within 
acceptable limits. 


Best Practices for Life-Cycle-Related CTE 
Assessment 


¢ Include experts on appropriate LCR 
areas on the independent TRA 
panel, preferably the same ones 
used in the CTE identification 
process 


¢ If the LCR CTE is also critical froma 
performance perspective, also 
determine a performance-related 
TRL 
— TRAs evaluate readiness to 
transition to the next phase of 
development 
— Therefore a// aspects of their 
maturity should be assessed 
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Outline 


¢ Introduction to Technology Readiness 
Assessments (TRAs) 

¢ Life-cycle-related (LCR) Critical Technology 
Element (CTE) identification 

¢ Life-cycle-related Critical Technology Element 
assessment 


References and Resources 


Defense Acquisition Resource Center 
http://akss.dau.mil/darc/darc.html 


— DoD Directive 5000.1 (DoDD 5000.1), The Defense Acquisition 
System, dated May 12, 2003 


— DoD Instruction 5000.2 (DoDI 5000.2), Operation of the Defense 
Acquisition System, dated May 12, 2003 


— Defense Acquisition Guidebook 
TRA Deskbook ( 
http://www.defenselink.mil/ddre/doc/tra_deskbook_2005.pdf y, 
DDR&E A. WA 
— Mr. Jack Taylor jack.taylor@osd.mil <i —— 
Institute for Defense Analyses 
— Dr. David Sparrow dsparrow@ida.org 
— Dr. Jay Mandelbaum jmandelb@ida.org 
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How TRAs Got Started 


“Program managers’ ability to reject immature technologies is | el 3 


hampered by (1) untradable requirements that force acceptance 

of technologies despite their immaturity and (2) reliance on —_ 
tools that fail to alert the managers of the high risks that would =~ 
prompt such a rejection.” GAO/NSIAD-99-162 


“Identify each case in which a major defense acquisition program entered 
system development and demonstration ... into which key technology has 
been incorporated that does not meet the technology maturity requirement ... 
and provide a justification for why such key technology was incorporated and 
identify any determination of technological maturity with which the Deputy 
Under Secretary of Defense for Science and Technology did not concur and 
explain how the issue has been resolved.” National Defense Authorization 
Act for Fiscal Year 2002 


“The management and mitigation of technology risk, which allows less costly 
and less time-consuming systems development, is a crucial part of overall 
program management and is especially relevant to meeting cost and schedule 
goals. Objective assessment of technology maturity and risk shall be a routine 
aspect of DoD acquisition.” DoD/ 5000.2, paragraph 3.7.2.2 


Stop launching programs before technologies are mature 


ESS eS 
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Quantifying the Effects of Immature 
Technologies 


According to a GAO review of 54 DoD programs: 


— Only 15% of programs began SDD with mature 
technology (TRL 7) 
¢ Programs that started with mature technologies averaged 
9% cost growth and a7 month schedule delay 
¢ Programs that did not have mature technologies averaged 
41% cost growth and a 13 month schedule delay 
— At critical design review, 42% of programs 
demonstrated design stability (90% drawings releasable) 
¢ Design stability not achievable with immature technologies 
¢ Programs with stable designs at CDR averaged 6% cost 
growth 


¢ Programs without stable designs at CDR averaged 46% 
cost growth and a 29 month schedule delay 


Source: Defense Acquisitions: Assessments of Selected Major Weapon Programs, GAO-05-301, March 2005 
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A 
G SAN U.S. General Services Administration 


Federal Acquisition Service 


Who are we? 


> GSA Federal Acquisition Service...A New Service 
Representing 
e General Supplies and Services 
e Integrated Technology Services 
e Assisted Acquisition Services 
e Travel, Motor Vehicle, & Card Services 


A 
G SAN U.S. General Services Administration 


Federal Acquisition Service 
Today’s Topic - Part One 


> Acquisition, Innovation and the Industrial Base 


> Enabling GSA “Tools” (Professional Services 
Schedule contracts) 


> GSA Tools — T&E Crosswalk 


> Innovation Examples 
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Today’s Topic - Part Two 


> The GSA Multiple Award Schedule Contract 
> Basic Schedule Ordering Procedures 

> Flexible Schedule Tools 

> GSA E-Tools 

> GSA Support 


A 
y XN 
U.S. General Services Administration 


Federal Acquisition Service 


NDIA’s Principle Missions 


eebeaes eee are to improve weapons technology, improve 
defense management, and maintain a strong science- 
industry-defense team continually responsive to all 
needs of the research, development, test & evaluation, 
production, logistics and management phase of national 
preparedness..... 
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Conference Objectives 


> T&E Metrics for Suitability and Sustainability 


> Reducing Total Ownership Costs and Role of T&E, 
SE, and Logistics 


> Test Planning 


> Planning and Implementing Sustainability as a 
KKP Effectively 


> Design Techniques 
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Conference Objectives, con't 


> Test Methodology 

> Testing for Realistic Estimates of Reliability 
> Technologies to Reduce Life Cost 

> Field Test Data & Archiving 


> Feedback Sustainment Lessons to Improve 
Requirements, Programming T&E and 
Acquisition Process 


S A 
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Leveraging the Industrial Base for 
Test & Evaluation 


> T&E Agencies 
> Acquisition 
> Industry 


GSA can be the enabler! 
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Commercial Item Acquisition 


Expanding the use of commercial items in DoD systems offers 
the DoD opportunities for reduced cycle time, faster insertion of 
new technology, lower life-cycle costs, greater reliability and 
availability, and Support from a more robust industrial base. It is 
a fact that for many technologies that are critical to military 
systems, the commercial marketplace-and not the DoD-now 
drives the pace of innovation and development. 


DoD Extension to PMBOK ® Guide 
June 2003 
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Acquisition Methodologies 


> Full and Open 
> Agency Organic Indefinite Delivery-Indefinite Quantity 
> Set-Aside Programs 


> Government-Wide Acquisition Contracts 
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Federal Acquisition Service 
GSA Schedules Program 
What is a Schedule? 


> GSA Awarded Competitive IDIQ contracts 
> Mirrors Commercial Buying Practices 


> Long-term Contracts Awarded to Multiple Companies 
-5 yrs. With Three 5-yr. Options 


> Forty-three Schedules Offer 11 Million-Plus Products & Srv. 


> Huge Selection: Over 14,000 Companies Represented 
on Nearly 18,000 Contracts 
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Multiple Award Schedule Features 


> 


VV V WV 


MAS Program Has Grown From $10 Billion in 1999 to over 
$35 Billion in 2006 


Service Schedules Pave the Way for Integrated Solutions 
Used Throughout DoD, DHS and Other Top Agencies 
Flexibility to Support Multi-Agency Strategic Sourcing Efforts 
Full Suite of Customized Online Tools 

e Air Force Advantage, Other Specialized Stores 

e e-Buy, e-Library 
Online Contract Terms and Conditions 


Spend Data Helps Agencies Manage More Effectively 
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Multiple Award Schedule Benefits 


All Competition Requirements Have Been Met 


> No Synopsis Required 
> Prices Have Been Deemed Fair & Reasonable 
> Terms and Conditions Have Been Pre-negotiated 


> Reduced Need For Front-End Procurement 
Personnel 


Continued 
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Multiple Award Schedule Benefits, con't 


> Reduced Procurement Lead Time 

> Minimizes Documentation Required 
> Customized Solutions 

> Direct Relationship With Contractors 
> No Additional Administrative Fees 

> Risk of Protest is Low 


CRY 
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Transparent Pricing 


Task Order Type: 


> Firm fixed price or FFP (preferred) 
> Labor hour vw 
> Time and Materials 
> Incentives with FFP 


The service contract act does not apply! 


GSA 


U.S. General Services Administration 


Federal Acquisition Service 


Enabling Tools 


GSA Professional Services 
Schedule Contracts 
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GSA Professional Services (samples) 


> Professional Engineering Services 

> Mission Oriented Business Integrated Services 
> Logistics Services 

> Environmental Services 


> Laboratory Services 
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Professional Engineering 
Services - Schedule 871 


= 
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Professional Engineering Services 
Scope of Schedule 


This schedule provides a 
comprehensive vehicle for 
Federal agencies to use 
when obtaining all types 


of engineering services 
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Professional Engineering Services 
NAICS Codes 


> 541330 Engineering services and 


> 541710 Research and development in the physical 
sciences 


U.S. General Services Administration 


Federal Acquisition Service 


Professional Engineering Services 
(Schedule 871) Industry Partners 


> Over 800 Contractors as of January 2007, 
including: 

e Large Businesses 

e Small Businesses 
— Disadvantaged 
— HUBZone 
— Veteran-owned 
— SDVOB 
— Women-owned 
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Professional Engineering Services-871 


> Special Item Numbers (SINs): 


¢ 871-1 Strategic Planning for Technology 
Programs/Activities 


¢ 871-2 Concept Development and Requirements 
Analysis 


¢ 871-3 System Design, Engineering and Integration 
¢ 871-4 Test and Evaluation 

¢ 871-5 Integrated Logistics Support 

¢ 871-6 Acquisition and Life Cycle Management 


GSA 


U.S. General Services Administration 


Federal Acquisition Service 


SIN 871-1 


> Strategic Planning for Technology 
Programs/Activities 


> ...definition and interpretation of high-level organizational 
engineering performance requirements such as projects, 
systems, missions, etc. 


GSA 


U.S. General Services Administration 


Federal Acquisition Service 


SIN 871-2 


> Concept Development and Requirements Analysis 


> ...abstract or concept studies and analyses, additional 
requirements definition, preliminary planning and evaluation of 
alternative technical approaches...cost-performance trade-off 
analysis, feasibility analysis.. 
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SIN 871-3 


> Systems Design, Engineering and Integration 


> ...translation of a system or (Subsystem, program, project 
activity) concept into a preliminary and detailed design 
(engineering plans and specifications), performing risks 
identification/analysis/mitigation, traceability... 


GSA 


U.S. General Services Administration 


Federal Acquisition Service 


SIN 871-4 


> Test and Evaluation 


> ...application of various techniques demonstrating that a 
prototype system (subsystem, program, project or activity) 
performs in accordance with the objectives outlined in the 

Original design.... 


S A 
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SIN 871-5 


> Integrated Logistics Support 


> ....analysis, planning and detailed design of all engineering 
specific logistics support including material goods, personnel, 
and operational maintenance and repair of systems throughout 
their life cycles..... 
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SIN 871-6 


> Acquisition Life Cycle Management 


> ...planning, budgetary, contract and 
systems/program management functions required to 
procure and/or produce, render operational and 
provide life cycle support to technology based 
systems, activities, sub-systems, projects, etc..... 
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Mission Oriented Business Integrated 
Services (MOBIS)-874 


> Special Item Numbers (SINs): 
¢ 874-1 Consulting Services 
¢ 874-2 Facilitation Services 
¢ 874-3 Survey Services 
¢ 874-4 Training Services 
874-5 Support Products 
¢ 874-6 Competitive Sourcing Support 
874-7 Program Integration and Project Management Services 
874-99 Introduction of New Services 
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MOBIS Schedule NAICS Codes 


> 541611Administration Management & General 
Management Consulting Services 


U.S. General Services Administration 


Federal Acquisition Service 


Some MOBIS Tasks 


> 
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Quality Management 

Business Process Re-engineering 
Strategic & Business Planning 
Benchmarking 

Competitive Sourcing 
Activity-Based Costing 


Financial Management Analysis 
(related to improvement effort) 


U.S. General Services Administration 


Federal Acquisition Service 


More MOBIS Task Examples 


> 
> 
> 


> 
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Statistical Process Control 
Surveys 


Individual & Organizational Assessments & 
Evaluation 


Process Improvements 
Process Modeling and Simulation 


Performance Measurement 
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Logistics Worldwide (Logworld)-874V 


> Special Item Numbers (SINs): 


¢ 874-501 Supply & Value Chain Management Services 
¢ 874-502 Acquisition Logistics 


¢ 874-503 Distribution & Transportation Logistics 
Services 


¢ 874-504 Deployment Logistics 
¢ 874-505 Logistics Training Services 
¢ 874-506 Support Products 


¢ 874-507 Operations & Maintenance Logistics 
Management and Support Services 


A 
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Federal Acquisition Service 


Logistics Worldwide Schedule 
NAICS Codes 


> 541614 Process, Physical Distribution, and Logistics 
Consulting Services 


> 561210 Facilities Support Services 
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Some LOGWORLD Tasks 


> Design & Fabrication (in support of a logistics 
effort/process) 


> System Testing 


> Range & Communications Engineering (But any IT 
must be related to logistics application/task. No volume 
purchases or software development not related to 
Logistics tasks.) 


> Remote site logistics support (National & International) 


> Food Service / Motor Pool / Courier Service 


U.S. General Services Administration 


Federal Acquisition Service 


More LOGWORLD Tasks 


> 


> 


Spares Support (to include purchase support if items are 
listed on GSA contract) 


Inventory Management 


Analysis of Distribution Points — Air, Road, Water, Rail or 
Pipeline 


Material Handling Training / Forklift Certification 


Planning (Scenarios, databases, after action review support, 
archive operational lessons learned) 


Train and mentor foreign military services in logistics methods 
and techniques 
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Environmental 
> Special Item Numbers (SINs): 
¢ 899-1 Environmental Planning Services & Documentation 
e 899-2 Environmental Compliance Services 
e 899-3 Environmental Occupational Training Services 
¢ 899-4 Waste Management Services 


¢ 899-5 Reclamation, Recycling and Disposal Services (This 
does NOT include handling/disposal and/or transportation of 
nuclear or radioactive waste.) 


e 899-6 Remote Advisory Services 
¢ 899-7 Geographic Information Services (GIS) 
¢ 899-8 Remediation Services 


S A 
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Environmental Schedule NAICS 


> 541620 Environmental Consulting Services 
> 562920 Material Recovery Facility 

> 562112 Hazardous Waste Collection 

> 541380 Lab Services [check scope] 

> 562910 Remediation Services 
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Some Environmental Tasks 


> Environmental Impact Statements 

> Develop programs or regulations 

> Risk Analysis or Vulnerability Assessments 

> Environmental Compliance Audits 

> Spill Prevention 

> Environmental Training 

> Permitting 

> Waste Management Plans or Studies 

> Establish/operate HAZMAT, electronics, CRT, battery or 
chemical recycling programs 
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Laboratory Testing and Analysis Services 
(Now part of Schedule 66) 


> Special Item Numbers (SINs): 


¢ 873-1 Mechanical Testing and Analysis 

¢ 873-2 Chemical Testing and Analysis 

¢ 873-3 Electrical Testing and Analysis 

¢ 873-4 Geotechnical and Thermal/Fire Testing and 
Analysis 

¢ 873-99 Introduction of New Testing and Analysis 


‘Chey 
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GSA Tools —- T&E Crosswalk 
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T&E Metrics for Suitability and 
Sustainability 


> SIN 871-6 Acquisition Life Cycle Management 
> SIN 874-502 Acquisition Logistics 
> SIN 871-5 Integrated Logistics Support 


> SIN 874-7 Program Integration and Project 
Management 
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Reducing Total Ownership Costs and 
Role of T&E, SE and Logistics 


> SIN 871-6 Acquisition Life Cycle Management 


> SIN 871-2 Concept Development and Requirements 
Analysis 


> SIN 871-5 Integrated Logistics Support 


> SIN 874-7 Program Integration and Project 
Management 
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Test Planning to Assure Priority for 
Assessment of S&S 


> SIN 871-3 Systems Design, Engineering and 
Integration 

> SIN 871-4 Test and Evaluation 

> SIN 874-1 Consulting Services 


> SIN 874-7 Program Integration and Project 
Management 
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Planning and Implementing Sustainability 
as a KKP Effectively 


> SIN 871-3 Systems Design, Engineering and 
Integration 


> SIN 871-4 Test and Evaluation 
> SIN 871-5 Integrated Logistics Support 


> SIN 874-7 Program Integration and Project 
Management 
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Design Techniques such as Conditioned- 
Based Maintenance and its T&E 


> SIN 871-3 Systems Design, Engineering and 
Integration 


> SIN 871-4 Test and Evaluation 
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Test Methodology 


> SIN 871-2 Concept Development and Requirements 
Analysis 


> SIN 871-3 Systems Design, Engineering and 
Integration 


> SIN 871-4 Test and Evaluation 
> SIN 874-1 Consulting Services (MOBIS) 


S A 
XN 
U.S. General Services Administration 


Federal Acquisition Service 


Testing for Realistic Estimates for 
Reliability 


> SIN 871-Test and Evaluation 


> SIN 874-7 Program Integration and Project 
Management 
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Reducing Total Cost of Ownership 


> SIN 871-1 Strategic Planning for Technology 
Programs 


> SIN 871-5 Integrated Logistics Support 
> SIN 871-6 Acquisition Life Cycle Management 
> SIN 874-1 Consulting Services (MOBIS) 
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Technologies to Reduce Life Cycle Cost 


> SIN 871-1 Strategic Planning for Technology 
Programs 


> SIN 871-2 Concept Development and 
Requirements Analysis 
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Field Test Data and Archiving 


> SIN 871-4 Test and Evaluation 
> SIN 871-6 Acquisition Life Cycle Management 
> SIN 874-7 Program Integration and Project Management 
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Feedback Sustainment Lessons to 
Improve Requirements, Programming 
T&E and Acquisition Process 


> SIN 874-1 Consulting Services (MOBIS) 

> SIN 871-5 Integrated Logistics Support 

> SIN 871-6 Acquisition Life Cycle Management 

> SIN 874-7 Program Integration and Project Management 
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INNOVATION Examples-CTAs, BPAs 


1. NASA Marshall Space Flight Center 
e One Schedule 
e One Task Order, One Contractor 
2. Marine Corps Systems Command 
e Eight Schedules 


ntractor Teams 


GSA 


U.S. General Services Administration 


Federal Acquisition Service 


NASA: One Schedule, One Task Order seo relseeeediacsta vi asaaiee: 


Aggregated Requirement 
eLogistic Support Services Logistics Worldwide 


¢Property (LOGWORLD) Schedule 874V 


hee eSIN 874-501 Supply & Value Chain Mgmnt 
eSIN 874-502 Acquisition Logistics 
“Disposal eSIN 874-503 Distribution & Transportation 
*Motor Pool . *SIN 874-504 Deployment Logistics 
*Equipment Maintenance *SIN 874-505 Logistics Training 
eEnvironmental eSIN 874-506 Support Products 
eFood eSIN 874-507 Operations and Maintenance 
Logistics Management & Support 


:B. Select Applicable SINs : 
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MARCORSYSCOM Requirements 


Multiple diverse program offices (each $20M+ annually) 
> Information Systems & Infrastructure 

> Battle Management/Air Defense 

> Infantry Weapons Systems 

> Armor & Fire Support 

> Transportation/Engineering Systems 

> Combat Equipment Support Systems 
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Solution: Multiple BPAs, Multiple CTAs 


> All requirements met through Schedules 
¢ $3.6B+ annually (supplies & products) 
e $180M+ annually (services) 


¢ $340M+ total GSA Task Orders (through May 
2005) 


¢ GSA Task Orders ranging from $150K to $6M 
each 


> 27 Contractor Teams, 27 Multiple-Award BPAs 
> 150+ Schedule contractors 
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Multiple-Schedule Solution 


Specialty Engineering Business & Analytical 


874 MOBIS 874 MOBIS 

871 Engineering 520 Financial & Business 
899 Environmental 69 Training 

70 IT 


Engineering & Scientific Acquisition, Log. & Admin. 


874 MOBIS 874 MOBIS 
871 Engineering Services 874 V LOGWORLD 
70 IT 871 Engineering 
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Multiple BPAs & Contractor Teams 


Specialty Engineering 


° FY05: 8 BPA Teams 
¢e Avg. Team Size: 11 
¢ Team Leads: AOT, AT&T, Battelle, 
CSC, EMA, ManTech, MTC, Unitech 


Engineering & Scientific 


e FY05: 8 BPA Teams 

¢ Avg. Team Size: 12 

¢ Team Leads: AERA/EDO, Anteon, 
BAE, DCS, NGMS, OSEC, Sverdrup, 
SAIC 


Business & Analytical 


¢FY05: 4 BPA Teams 

¢ Avg. Team Size: 8 

¢ Team Leads: Booz-Allen, Kalman, 
MCR, RCI 


Acquisition, Log. & Admin. 


-FY05: 7 BPA Teams 

¢ Ave. Team Members: 11 

¢ Team Leads: CACI, EG&G, INS, 
MKI, Titan, BRTRC, CRC 


A 
XN 
U.S. General Services Administration 


Federal Acquisition Service 


Examples: Using CTAs & BPAs for 
Complex Requirements 


> Marine Corps Systems Command 
www.marcorsyscom.usmc.mil/sites/acss/ 


> Army Aviation and Missile Command 
https://wwwproc.redstone.army.mil/acquisition/omnibus2 


A 
G SAA U.S. General Services Administration 


Federal Acquisition Service 


PART TWO 


> The GSA Multiple Award Schedule Contract 


> Basic Schedule Ordering Procedures 
> Flexible Schedule Tools 

> GSA E-Tools 

> GSA Support 
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Multiple Award Schedule 
Contract 
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Schedule Contract Features 


> Streamlined Ordering 

> Best Value 

> Blanket Purchase Agreements 
> Teaming Arrangements 

> Maximum Order Provisions 

> E-Tools 
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Streamlined Ordering 


> Under $3,000 
e Select any Schedule Contractor 


(The Micro Purchase Threshold for Davis Bacon Act Procurements is $2,000. 
The Micro Purchase Threshold for Service Contract Act Procurements is $2,500.) 


> Over $3,000 
e Send RFQ to a Minimum of 3 Schedule Contractors 
e Review the Quotes Received (can use GSA e-Buy) 
e Select the “Best Value” Based on Other Evaluation Factors in 
Addition to pricing 
e Issue a Task Order/Delivery Order 
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Awards Based on Best Value 
Best Value is Determined on What is Important to the Buyer 


> Price 

> Technical Solution (including program management) 
> Corporate Experience 

> Past Performance 

> Delivery 
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Blanket Purchase Agreements 


> BPAs Offer Flexible Solutions and Options 
> Build Long Term Relationships With Vendor Partners 
> Useful for Replacing Expiring IDIQs 
> Easy to Establish 
e Establish BPA From the Pool of Schedule Contractors 


¢ Used to set up “Accounts” to Fill Recurring 
Requirements 


e Can Last as Long as the Contract Period 

e Review Annually for “Best Value” 

e May be Offered Volume Discounts When Establishing 
a BPA 
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Contractor Teaming Arrangements 


> Schedule Contractors May Team With Each Other to 
Provide a Total Solution for a Customer 


> Best for Complex Acquisitions 
> Enables Contractors to Consolidate Unique Capabilities 


> Offers the Government the Best Combination of 
Performance, Cost and Delivery 


> Increases Small Business Participation 


> CTA Team Leader Responsible for Task Order Execution, 
Even If Not Identified 
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Maximum Order Provisions 


> Customer Agencies Must: 
e Seek Additional Price Reductions 
e 3+ Contractors for Further Competition 


> Contractors can: 
e Offer a Lower Price 
e Offer the Current Contract Price 
e Decline the Order 
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GSA Schedule Contract Fundamentals 


> Standing solicitation - offers accepted throughout the year with 
no closing date (uniform updating, mass mods conform all 
Schedule contracts to existing solicitation) 
> Part 12 Multiple Award IDIQ (FFP wW/EPA) 
> Task Orders (FFP, LH, and/or T&M) 
> Three 5-year option periods 
e Task Order performance could extend beyond since 
performance continues even if Schedule option not 
exercised (or contract terminated) 
¢ However, agency couldn’t award new orders or exercise 
options on TOs/BPAs without existing Schedule contract 
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Pricing & EPA Methods 


1. Established Commercial Price List/Equivalent 


Requires a Modification Request from Contractor for 
Increase to Take Effect 


2. Negotiated Escalation Prior to Award 


A. Fixed Escalation for Term of Contract (Multi-Year Pricing 
20 Years) 
Price Increases Automatically Effective on the 
Anniversary of Contract (No Modification) 

B. Adjustments Based on Market Indicator (Yearly Mod) 
Based on a Published Index, Survey or Market Indicator 
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Are Schedule Prices the Best Prices? 


> Contract Level 
¢ Contract Pricing Objective: “Most Favored Customer” 
e Price Reductions Clause 
e Inspected during contractor assist visits 
¢ Ordering agencies pay no fee to GSA to use the Schedules 


> Task Order Level 
e Shall seek discounts (FAR 8.405-1(d)) if: 
¢ Task Order exceeds “Maximum Order Threshold” 
¢ BPA regardless of estimated value 
¢ Encouraged to seek discounts for all orders 


e Make best-value determination, total price/LOE/labor mix 
reasonableness 
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CONTRACT TERMS and CONDITIONS 


> What Does the Schedules Contract Say? 


> Where Can I Read the Contract Clauses? 
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Search FedBizOpps Schedule RFP 


> Standing solicitation periodically replaced (GSA says 


“refreshed”) 


All contracts updated with mass modifications to 
match the most recent FedBizOps-posted “refresh” 


Opens as a searchable Word document 
Use <Edit> <Find> in Word to locate words or 
phrases of interest. 


Good method both for numbered clauses and for 
other “free text” 
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Schedules Onre ry GSA Advantage 


Federal Supply Schedule Listing Basic Schedule Ordering Guidelines 


Search: [all the words 7 | 


Schedule Summary 


For general schedule questions, contact: 


Phone: 1-800-241-RAIN 
E-mail: environmental@gsa.gov 


899 ENVIRONMENTAL SERVICES 


Federal Buyers... Click here to view the current 


mn Schedule 


Supplies & 
Services 


View Contract Clauses >> solicitation on Fed BizOpps 


899 Category list: [- Categories - | 


Category Description 


g99 1 Environmental Planning Services & Documentation - Services to include, but not limited to: Environmental Asses: 
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BASIC SCHEDULE ORDERING 
PROCEDURES 


& 


TASK ORDER CONTRACTING 
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In-Scope or Out-of- 
Scope? 


Out-of-Scope 
«Non-Commercial 


eConstruction/A&E 
eCost-type contracts 


GSA 


Schedule 
Scope 


GSA 
Contract 
Scope 


eTerms & Conditions conflicting 
with Contract (or Schedule) 

eServices outside the scope of 
the Contract (or Schedule) 


Task Order 
Scope 


Out-of-Scope 
«Degree of Scope Changes 


eType of Scope Change 


Modification 
Scope 


eAmount of price or period 
of performance change 
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Some Limitations 


> Broad Acquisition Limitations 


e Commercial Services 

e Personal Services 

e Inherently Governmental Functions 
> Schedules Program Limitations 

e Architect/Engineer Contracts 

e Cost-Reimbursement Type 

e Construction 
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Ordering Procedures & Issues 


> Schedules as Priority Source 

> Ordering (RFQ Process, Quotes, Evaluation) 
> Types of Orders 

> GSA Orders vs. Open Market Procurement 
> “Other Direct Costs” 
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The Basic Schedule RFQ Process 


> Requirements Identification 

> Market Research 

> Acquisition Planning 

> Develop Performance Work Statement 

> Develop & Distribute RFQ (w/ Selection Factors) 
> Evaluate Quotes Received 

> Task Order Award 
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Prepare & Distribute RFQ 


> Select non-conflicting order clauses 


> FAR Clauses for Commercial Items [in GSA 
Schedule contract already] 


> Define quote submittal requirements 

> Establish offer/quote deadline 

> Obtain necessary approvals (AAS, etc.) 

> Provide to Schedule holders (e-mail, eBuy, fax) 
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Contents of RFQ for GSA Task Order 
(Keep It Simple) 


> What Does the Government Think is Important for 
Award? (Evaluation Factors) 

> What Does the Government Need to See? (Quote 
Submittal Instructions) 

> What Are the Order’s Terms & Conditions (if any) 
Not Already in the Schedule Contract? (add non- 
conflicting clauses) 


> What Will the Contractor Do? (PWS) 
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GSA Ordering 


>8.404(a) — Orders placed against a MAS using the 
procedures under this subpart are considered to 
be issued using full and open competition 


> Micropurchase: Place orders with any Schedule 
contractor, but should “rotate” buys. 


> Micropurchase to Maximum Order Threshold 
(MOT): Provide RFQ (PWS + Eval Factors + 
Submittal instructions) to at least three contractors 


>» Above MOT or if establishing BPA: As above plus 
an “appropriate number of additional contractors” 
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Receive & Evaluate Quotes 


> Oral Proposals? 
> Clarification of Minor Irregularities & Errors 
> Technical Review 
> Past Performance Review 
> Communications (“Negotiation”) 
e - Discount from GSA Schedule Price 
e -Reasonableness of Labor Mix/Total LOE 


A 
< SA U.S. General Services Administration 


Federal Acquisition Service 


Best Value Evaluation 


> The expected outcome of an acquisition that, in the Government’s 
estimation, provides the greatest overall benefit in response to the 
requirement. 


> Best Value Continuum: From price predominates (Low-Price 
Technically Acceptable) to technical/past performance 
predominates 


> Best Value permits tradeoffs between price and non-price factors. 
The ordering activity may be willing to pay more for: 


e Achieving Socioeconomic Objectives (but not “Set Aside” Task 
Orders) 


¢ Better Past Performance 
e Better Technical Approach 
¢ Better Management Capability 
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Level of Effort Task Orders 


> Time & Material [FAR 16.601] 
e Direct labor at fixed, hourly, fully-burdened rates 
e Materials at cost and handling costs 

> Labor Hour [FAR 16.602] 
e Like T&M, but contractor supplies no materials 


> Fixed Price is preferred over LOE 
e Like Cost-Reimbursement, no contractor incentive for cost control 
e Requires more surveillance and control than FFP 
e Use only where duration and extent of work cannot be estimated 
e CO determination, sometimes higher approval, required 
e Include ceiling price contractor exceeds at its own risk 
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GSA Orders (vs. Open Market) 


> No FedBizOps synopsis for GSA Orders 

> Prices on FSS Authorized Price List (GSA web-posted) 
already determined fair & reasonable, but Ordering 
Officer determines Best Value and reasonableness of 
overall LOE and labor mix. 

> Competition (CICA) requirements already met, no 
competing outside Schedules 

> Unrestricted (not “Set Aside” but can be eval factor), 
except for certain Schedules/SINs set aside by GSA 
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Other Direct Costs (ODCs) 


> Three Categories of ODCs 
1. Contract Support Items (on Schedule contract) 


2. Open-market items (“Incidentals”) (not on Schedule 
contract) 


3. Reimbursables: Lodging, Transportation & Per Diem 
All ODCs must be within the scope of the contract and awarded SIN 
ODCs support, are not the primary purpose of the order 
ODCs may not duplicate costs already included in the contract price 


ODC prices must be determined fair and reasonable by a CO (Schedules level or 
Task Order level) 


VV VV 
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1. Contract Support Items 


> Commercial Items 
> Items included in the MAS contract 
- Contract award 
- Added by Schedule contract modification 
- Not to be separately ordered without the services 
> Items for which the Schedule CO has already determined the 
price fair and reasonable 
> Example: items awarded, priced, and listed under a “Support 
Products” SIN (but separate product-only SIN not required) 


A 
G SAA U.S. General Services Administration 


Federal Acquisition Service 


2. Open-Market Items («ncidentats”) 


> Items not awarded under that Federal Supply Schedule contract 
> No assumption of price reasonableness (or scope) 
> Schedule T&Cs don’t apply unless cited 
> Open-market items purchased IAW all applicable acquisition regulations: 
e FAR Part 5 — Publicizing contract Actions 
e FAR Part 6 — Competition Requirements 
e FAR Part 12 — Acquisition of Cl’s 
FAR Part 13 — Simplified Acquisition Procedures 
FAR Part 14 — Sealed Bidding 
e FAR Part 15 — Contracting by Negotiations 
e FAR Part 19 — Small Business Programs 


A 
G SA U.S. General Services Administration 


Federal Acquisition Service 


3. Travel Reimbursables 


> Federal Travel Regulation 
e 41 CFR, Chapters 300 — 304 
e Travel policies for Federal civilian employees and 
others authorized to travel at Gov’t expense 
> Joint Federal Travel Regulations 
e USC, Title 37 and 10 
e Availability of contract fares or prices to 
government contractors 
> Local travel in the performance of a task order 
e Reimbursable IAW ordering agency regulations 
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FAR Parts Not Applicable to Schedule 
Orders/BPAs 


> Synopsis requirements in Part 5 

> All of Part 6 (8.405-6 is “Limiting Sources”) 
> All of Part 13 (except 13.303-2(c)(3)) 

> All of Parts 14 and 15 

> All of Part 19 (except 19.202-1(3)/(ili)) 


However, if mixing open-market items (total over 
micropurchase) then all regulations apply to those items. 
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“Limited Source Justification” 
GSA Sources 


> Only one source capable of responding due to the unique or 
specialized nature of the work 


> New work is a logical follow-on to an existing order (excluding 
previous orders placed previously under sole source 
requirements) 


> Item is peculiar to one manufacturer (a brand name item, 
available on various Schedule contracts, is an item peculiar to 
one manufacturer); or 


> An urgent and compelling need exists and following the 
ordering procedures would result in unacceptable delays 
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How Can Need for Quoted Open Market 
Products Be Reduced? 


> Contractor requests GSA CO to add to contract by 
modification 

> Agency preference for all-Schedule order (Schedule 
contractors team) 

> Agency procures and furnishes as GFP/GFE (need 
those clauses in Task Order) 
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Pricing in Task Order Quotes 
> Compare with website GSA Advantage price list 
> Scrutinize non-Schedule services & items 
> Labor rates on Schedule contract are already fully- loaded 
> Question additional G&A, esp. on travel (paying twice?) 
> 


Don’t pay a separate “Industrial Funding Fee” (that 0.75% already 
included in contract price) 


Vv 


If concerned about what is included in price, contact the GSA CO 


V 


Contract labor rates assume “normal” bid & proposal expense - - 
may get “no quote” if unusual quote requirements for Task 
Order/BPA 
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Flexible Schedule Tools 


> Blanket Purchase Agreements (BPAs) 


> Contractor Teaming Arrangements (CTAs) 
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Blanket Purchase 
Agreements 
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Why Establish MAS BPA? 


> Satisfy recurring requirements 

> Reduce administrative burden 

> Leveraging buying power through volume 

> Support field offices/other contracting offices 

> Quicker order turn-around 

> Can incorporate non-conflicting terms & conditions 
> Can include contractor teaming 

> No funding required to establish BPA 


> No synopsis, no competition outside Schedules to 
establish or use BPA 


> Opportunity to negotiate better discounts 
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Blanket Purchase Agreements 


¢ Simplified Acquisition Method 
¢ Anticipated Repetitive Needs 
¢ Qualified Source(s) 

e Single or Multiple Award 


The Contractor's Perspective The Government’s Perspective 


eRecurring source of orders ¢Flexibility in exact services, quantities, 


¢Volume and period of performance 
eQuicker turnaround on orders eEfficiencies by having summary 
eExpectation of price discounting invoicing and consolidated payment 


eExpectation of best pricing for each 
Task Order 
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What’s ina BPA? 


> Estimated value/level of effort (not a ceiling) 
> Duration 

> GSA Schedule(s) & Contract(s) 

> Participating offices/agencies 

> Invoicing/billing procedures 

> Ordering procedures (if multi-award) 
> Terms & conditions 

> Discount terms 

> Types of orders to be placed 

> Scope 

> BPA Termination 


GSA 
XN 
GsA U.S. General Services Administration 


Federal Acquisition Service 


BPA Decision 
Multiple-Award BPA 
GSA | Determine 
Schedules In-Scope Yes 


No Yes 


Broad 
Requirement? 


Issue Recurring 


Tasks? 


RFQ/Task Order 


Quotes | be 


Single-Award BPA 
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Single-Award BPA 


Receive & Evaluate BPA Quotes 
Award One BPA 
Issue BPA RFQ (PWS) 


VV V 


Then, for Each Recurring Task: 


Award Task Order 

Issue Task Order RFQ (PWS) to sole 
Evaluate the Quote 

BPA-holder 


VVVV 


A 
G SAN U.S. General Services Administration 


Federal Acquisition Service 


Multiple-Award BPA 


1. Issue BPA RFQ (PWS) 


2. Award Multiple BPAs 
3. Receive & Evaluate BPA Quotes 


Then, for Each Recurring Task: 


1. Develop Task Order Evaluation Criteria 
2. Issue Task Order RFQ (PWS) to all 


BPA-holders 

3. Evaluate the Multiple Quotes: Best-Value 
Source Selection 

4. Award Task Order 


Two Best Value “Competitions”: Who Gets the BPA? Who Gets the Task Order? 
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Contractor Teaming Arrangements 
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Subcontracting vs “Teaming” 


> Only Prime must have a 
Schedule contract 


> Only Prime has privity of contract 
(and interface) with Government 


> Ordered and invoiced at Prime’s 
Schedule rate (less discount) 


> Limited to SINs and labor 
categories on a single Schedule 
contract 


> Prime can’t “delegate” 
responsibility 


> 


> 


> 


Each Team Member must have a 
Schedule contract 


Each Team Member has privity of 
contract (and can interface) with 
Government 


Ordered and invoiced at each 
Team Member’s Schedule rate 
(less discount) 

Total Schedule solutions possible 


Each member can be responsible 
for duties in a teaming agreement 
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Prime/Subcontractor Relationship 


Prime 
Contractor 


BPAs/Orders Only to Prime (Schedule Contractor) 
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MAS Teaming and Subcontracting 


Team Lead 
Team Team 
Member 1 Member 2 
S 
(has Schedule) Cs» 


Schedule Teaming Requires Teaming Agreement 
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MAS CTAs 


VVV VV VV WV 


Team Leads & Members must have GSA Schedule and use 
their Schedule rates 


Contractor Teams are issued one BPA 


Could include subcontractor effort, as long as mapped to their 
Prime’s Schedule labor category 


lf Multi-Award BPA, Teams compete for Task Orders 


Task Orders can be issued to Team Lead or directly to Team 
Member 


Not a separate legal entity but acts /ike joint venture 
Include Teaming Agreement with quote for agency review 
Government incorporates CTA into BPAs/Orders 


CTA Can Reduce Need for Open-Market Items on a GSA Task Order! 
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Teaming Agreement Highlights 


Identify Parties (Members and Lead) 
Teaming Activities (w/ responsibilities) 
Type & Duration of Agreement 

CTA Terms 

Ordering Procedures 

Team Lead & Team Member Duties 
Pricing, Invoicing, and Payment 
Performance Responsibility/Evaluation 
Reporting Sales to GSA 

Warranty 

Confidential Information 


VVVVVV VV VV WV 


Agreement is solely between the Members, can't conflict with Schedule 


GSA 


U.S. General Services Administration 


Federal Acquisition Service 


CTAs and BPAs: Putting It All Together 


BI’ “A 
Aggregate eUse Single/Consolidated 
— Schedule 
¢ Task Award to a Single 


Contractor 


Your Requirements 


«Use Several Schedule 
eAward BPAs 
«Contractors Team 
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Two Ways to Satisfy Multi-Domain 
(Schedule/SIN) Requirements 


1. One GSA Contractor holds all needed domains, so 
teaming not required: 


a. AS separate single-Schedule contracts, and/or 
b. On the Consolidated Schedule 


2 
7 


2. GSA Schedule holders team across domains. 
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Consolidated Schedule 


> Only for contractors with two or more Schedules 
(e.g., PES and MOBIS) 


> One GSA contract, so just one Task Order for agency 
to award/administer 


> Contractor can offer their entire business line ona 
single contract 


> Includes most service Schedules and some related 
product Schedules (including IT) 
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E-Tools: Bringing Buyers and Sellers 
Together 


> GSA Advantage!® 


> e-Buy 


> e-Library 
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Market Research Tools 


GSA Schedules mage 


GSA Federal Supply Service 


e Research vendors 
Buyer 


Advantage: <a> Hle-Buy 


¢ Research vendors ¢e RFQ/RFI 
¢e Research pricing ¢ Obtain quotes 
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Market Research Web Address 


Schedules E-Library 
www.fss.qsa.gov/elibrary 


GSA Advantage!® 
www.gsaAdvantage.gov 


GSA e-Buy 
access through GSA Advantage!® 


www.gsaAdvantage.gov 
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GSA Support 
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Support Provided By GSA 
> Agency outreach: guidance, training & education 


> Industry outreach: promote, facilitate the growth of 
schedules supplier base 


> Expedite awards/mods of “potential” suppliers 
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GSA Support (continued) 


Value added assisted procurement service 
or agency in-house buying are customer 
options. 
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Assisted Services Support Provided 
By GSA 


> Acquisition management 
> Program management 


> Financial management 
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Costs to Use GSA 


> No additional 
administrative fee for in- 
house buying (IFF built into 
rates) 


> Competitive negotiated fee 
for procurement service 


U.S. General Services Administration 


Jeff Manthos 
(703) 605-2838 
ffrey.manthos@gsa. 
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The Challenge 

a ee ee ee ee ee es 2 2 
¢ To provide the warfighter with the tools needed 

to win the current “Long War” while building a 

force to meet future threats — 

e Do more 

¢ Do it faster 

¢ Do it with fewer resources 

¢ Do it with an understanding of the true cost 


History 


e Suitability is a long standing challenge — often 
the poor cousin of effectiveness 
e There have never been any “good old days” 


e Aggregation of distinct but related disciplines 
¢ Reliability, Maintainability, Availability, Logistic 
Supportability 
¢ Compatibility, Interoperability 
¢ Documentation, Training, 
¢ Security, Information Assurance 
e¢ Safety, Human Factors 


How Are We Doing? 
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Technology 


e Technological advances provide opportunities 
and challenges 


Electronic technical manuals 

Improved human-machine interfaces 

Higher Order Languages & re-usable software 
COTS/GOTS components 

Open Architecture 

Speed to market 

Increasing complexity 


Cultural Issues 


e Acquisition system favors successful 
demonstrations of technology vice a rigorous 
assessment of potential manufacturing 
challenges 


e Early discovery is as likely to be penalized as 
rewarded 


¢ Focus on delivering new capability without 
understanding that ownership costs may result in 
less overall warfighting capability 


What can we do? 


¢ Develop an enterprise approach that eliminates 
traditional distinctions between acquisition and 
life cycle costs. 


OPERATING CONCEPT 


Oe od =O) od = p 
RESOURCES AT * DOLLARS \) 
EACH ENTERPRISE * STUFF 
INTERSECTION 


¢ CAPABILITY 


PROVIDERS 
& ENABLERS 


OUTPUT = 


READINESS 
COST 


INSTALLATIONS 


HEALTH 


DESIRED NAVY ENTERPRISE 
OQUTPUT 


Y READINESS OVER COST TODAY 
Y READINESS OVER COST TOMORROW 


Y READINESS OVER COST IN THE FUTURE 


Achieved Through Behavioral Model (Interdependent Concept of Operations): 


« Navy Enterprise (Governance Board): 


¢ Senior Navy strategic decision forum focused on improving productivity for current and future 
readiness through integration of supported Warfighter Enterprises. 


¢ Warfighter Enterprises (Five Supported Teams; Led by Super TYCOMs): 


¢ Collaborative teams focused on delivering warfighting capability to Navy Components and 
Combatant Commanders; and increasing productivity across their Domain at reduced cost. 


¢ Providers/ Enablers (Supporting Elements; with Designated Leads): 


¢ Operate as providers/ enablers to manage value streams (people, dollars, and stuff), supporting 
TYCOM-led Warfighter Enterprises, with linked and common processes/ metrics. 


. Domain: Dollars, people, & stuff associated with each Warfighter Enterprise. 

. Demand Signal: Derived from the Warfighter Enterprises (I.e., Readiness required and no more). 
¢ Entitlements: What's needed, when, how much, and no more. 

¢ Output: Readiness over Cost. 


What can we do? 


¢ Develop an enterprise approach that eliminates 
traditional distinctions between acquisition and life cycle 
costs. 


e Increase early involvement of the OT community 
e Early involvement efforts have tended to focus on mission 
effectiveness 
e There have been notable successes in identifying risks to 
maintainability, compatibility and safety 
e Make T&E a true element of systems engineering 
e Evaluators must provide timely feedback in a manner that does 
not place the program at risk 
e Developers must value inputs 


Leverage Modeling & Simulation 


e Exploit technological advances to develop high 
fidelity physics based models 
¢ Gain insights earlier in development 
e Assess performance in operationally realistic 
environments that cannot be replicated in actual test 
due to numbers of assets/security concerns (Self- 
defense test ship, Weapons Analysis Facility) 
e Leverage industrial techniques to understand risk 
areas in the manufacturing processes 


e Understand the limitations of the simulations 
employed particularly in areas such as 
compatibility and interoperability 


Suitability translates directly into 
combat power 


~ 


Fr 
COMOPTEVFOR Acquisition Community 
* Focus on Suitability ¢ Focus on optimizing the Enterprise 
investment 
¢ Increase technical expertise to > 
more rigorously assess RM&A eUnderstand total ownership costs 


*-Promote transparency 
e Leverage SYSCOM warrant ; ; 
holders for technical disciplines °Promote a systems engineering 
— approach 


ne to tee *Resist temptati on to < 
| suitability market buy-ins 


¢ Metrics - move from 
output 


() SOEING: 


Washirigton, D. Ct Operationsm™ 


Phantom Works / 1DS-AS 


Business Development / Government Relations 


Past, Present, & Future Trends in RM&A 
March 14, 2007 
23'4 Annual T&E Conference 


Ronald (Mutz) Mutzelburg 


Past Trends 
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e Late 60s: AF Advanced Logistics Systems 

e Mid 70s: AFLC Weapon System Budgeting 

e 1976: AF Acquisition Logistics Division Formed 
e Late 70s: RIW (Like PBL of Today) 

e Early 80s: The AF Great Engine War 

e Late 80s: LO Maintenance Challenge 

e« Late 80s: Air Staff GO Dedicated to RM&A 


¢ 90s until Today: RM&A Treated Equally With Other 
Performance Parameters 


Present Trends 
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e F-18 as an R&MA Trail Blazer 
e JSF is also Focused on RM&A 
e The Promise of UAVs 


«Proving Operational Suitability Always a 
Challenge 


¢Must Meet Depot 50/50 Rule Vice CLS & PBL 
¢ Antiquated IT 


mUisela-m Ba-ale 
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e Energy Efficiency 

e Net Centric Logistics 

e Enhanced Readiness Modeling & Analysis 
¢ Health Management, Diagnostics, CBM+ 
e Government-Industry PBL Teaming 


¢ Weapon System Reliability Equivalent to the Family 
Auto 


¢ Color of Money Challenge 


.c Acquisition 
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Outline 


Sustainment in the ARMY 

— The Need for Change 

— Impacts 

— Testing 

— How To Enable Success 
¢ Reliability 
¢ Maintainability 
¢ Sustainability 
¢ Testing 
¢ Policy Changes 

- Challenges 

- ATEC Support of GWOT Fielding 


ISSUE 


Army Systems Consistently Fail To 
Achieve Reliability Requirements 
During Operational Testing! 


The Need for Change 


46000 Demonstrated Reliability Versus Requirements for 
Operational Type Tests* © 


@o ® 
L 


-® 


i) 


Only 34% Met 


| 
Ha 4000 Requirement 
: A om 
3 s00 
% 600 = = — : 
5 -" 
= maa 
o 400 = * Causes of Failure: 
200 pk 5 = xd - Unrealistic Requirements 
Some” “ae * - Poor Planning 
Oo a =8 - Insufficient DT 
- Cost & Schedule 
4/14 7/06 


(Data 1996 - Oct 05) Requirement MTB_ 


*Continuous Systems 


Amongst Systems Which Did Not Meet Reliability Requirements In OT 
75% Of Them Failed To Achieve Half Of Their Requirement 
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Impacts 
Distribution of Life Cycle Costs 


(Percentages) 


Type System R&D Investment 


" Affects Warfighter 
- Mission Success ss 
- Ao & Force Effectiveness Surface sips 
- Log Footprint Jet fighters 
- Safety & Soldier Confidence 
" Life Cycle Costs (LCC) 
- Sustainment accounts for 60% LCC 
- Spares & Maintenance contribute to 
90% of Sustainment Costs — Reliability 
is the driver. 
- Reliability impacts billions of $$$ 


| We need a process to improve Sustainability for Army Systems 


Sources: Selected Acquisition Reports, VAMOSC systems 


Reliability ls Not Just About Testing wy 


Why? 

- accelerated acquisitions & immature 
designs 

- more complicated systems 

- software intensive systems 

- fielding of system of systems 

And 
Increased emphasis on Sustainability 


_ We ave tao things etre. 


Reliability Enablers 
(Embedded in FCS Program) 


- . Design In Reliabilit E : 
Physics of Failure (PoF) Improved Ribbon Bndge 


aT 
» Early Design Influence through Physics- Based 
Modeling of Critical Component Designs 
° Model the root causes of Failure ( Fatigue, pe. 
Fracture, Corrosion & Wear) | Pa 
Sys. Req: 104,520 Crossings 
PoF Usage 
¢ Avoided lengthy retest- saved $1-2M 


¢ Supported Urgent Materiel Release 
¢ IRB deployed 


Tangible benefits of PoF 


Reliability Enhancement Testing (RET) 


*Army Materiel Systems Analysis Activity 


> Used to Find Weak Areas of the Design and Correct 
them Before Production Begins 


> Destructive test which incrementally increases 
environmental stresses to determine Failure Modes - 
Find Design Faults 


Emphasize Early ID & Elimination of Failure Modes 


Maintainability Enabler: The Pit Stop Philosophy 
(Embedded in FCS Program) 


Pit Stop Design ts an approach for designing maintainability in military systems that 
is derived from auto racing. Emphasis is on simplicity of design to minimize downtime 
due to repair. Pit Stop Design is not designing to minimally meet a requirement, 
it is designing to win 


Some Characteristics of the design approach are: 

¢ Design component packaging minimizing weight 
and volume 

¢ Modular design to allow commonality and upgrades 

¢ Reduction of tools, with none being the goal 


¢ Service by a single soldier, whether a 1st percentile 
female or a 99th percentile male, in MOPP IV or 
winter gear 


¢ Handles and grip areas on components 
¢ Simplicity of design minimizes maintenance training 


Designs for Maintainability 
And Producibility 


Planning, Tracking & Verification 
Reliability Use Case* 


Evidence includes 3 objectives: 

1. Understanding the Requirements 

2. Planning & Implementation 
necessary to satisfy requirements 

3. Assurance that the requirements 
are being met 


*Contract deliverable for each FCS vendor 


Examples of Types of Evidence 
Y Design M&S 


Vv RAM Modeling, Component, 
subsystem, system testing 

vY Environmental Stress Analysis 

Y Structural Finite Element 
analysis 

v Fatigue Analysis 

v FMECA & Fault Tree Analysis 

v Prognostics Assessments 

v Rel. Enhancement Testing 

v Pit Stop Engineering 

Y Software Reliability 


Conduct Smarter Logistics Demonstrations 
— Implement Readiness Reviews prior to test conduct 
— Make more operationally realistic 


Apply Systems of Systems Approach To Sustainment 
Concepts 


— Eliminates stove piping & reduces cost and logistics 
footprint 


Incorporate Prognostics As It Matures 
— Big potential for gains in operational availability 


Apply Greater Use Of Sustainment Based M&S 
— Supports more accurate sustainment estimates 


Implement the Common Logistics Operating Environment 
(CLOE) 


— Enables real time visibility of sustainment requirements 
for fielded systems 


Why Integrated System Testing? 


¢ Technical/Developmental Testing Conducted 
- to find faults, implement corrective actions, and mature the design 
- to confirm technical capabilities/functionality and 


manufacturability. ATEC Studies Indicate Failure To Meet 


RAM Requirements In DT Almost Always 
Results In Failure In OT 


¢ Operational Testing Conducted 
- to provide info. on integration of the Soldier, the support system, 
training & doctrine, and materiel in an operational environment. 

- to confirm/demonstrate operational suitability 
requirements. 


Need Earlier Testing 
With Operational 
Realism 


The Costs Of Operational Testing And The Importance Of Correctly Deciding 


Whether To Proceed To Full Rate Production Makes It Important To Base Decisions 
On All Available Relevant Information! 


RAM Operational Testing Concemswiy 


¢ Difficult to get sufficient sample in OT to address RAM 
- Cost, schedule, soldier availability, data collection 
on non-interference basis difficulties 


e Leverage DT to address Reliability requirement 
compliance 


- Historical data indicates no difference in DT-OT 
reliability performance for: AV, CMBT. Vehicles, 
Automotive, AMMO, Soldier & Combat Support Items 

- OT does not have to be “sized” for RAM. Can provide 
info on induced operator/maintainer failure modes 


- For electronics, DT not a good predictor of OT 
Reliability performance 
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Requirements & Policy Changes 


Mandatory Sustainability KPP for all JROC programs along with two Key System 
Attributes (KSAs) 


- KPP - Availability 
- KSA - Materiel Reliability 
- KSA - Lifecycle Ownership Costs 


(TRADOC Reg. 71-20) 


DOTE has emphasized the Need for Change/Improvement in Reliability/Sustainability 


Type Classification-Standard and Full Materiel Release now required for FRP 
- Sustainability & RAM requirements must be met 


Army T&E Acquisition Executive Initiative 
- Establishment of Reliability Thresholds early in SDD phase 


Challenges 


w 


¢ Need To Design Reliability And Maintainability Up-Front — Minimize Log 
Footprint And Associated Costs 


¢ Need To Force The Contractor To Plan For R&M Maturation And RAM 
Requirement Compliance 


¢ Ensure System Of Systems Approach Is Applied To The Sustainability 
Concept 


¢ Need To Better Utilize All Relevant Data In Assessing RAM And 
Sustainability 


- Benefit through early testing with operational realism 
- Integration of developmental and operational testing 
- Better use of Modeling and Simulation to address Sustainability 


“We've already run this through our Abacus model and we 
can save 900 Lira by not taking soil tests.” 


Direct Support to GWOT by ATEC wy 


Detailed Brief by Brian Simmons (AEC Director) on 
Thursday 


Normal Test Process Did Not Meet Urgency Timeline 
Soldiers and Commanders Still Need to Know: 

— Does It Work? 

— How Do | Know? 

“Leaned” T&E Processes 

— SER > Capabilities and Limitations Report 

Long Term Reliability Remains a Concern 


Joint Advanced 
Warfighting Program 


} 


Suitability Assessment 
in 
Capabilities-Based T&E 


Vincent P. Roske, Jr. 
Institute for Defense Analyses 


3/16/2007 


Poon Suitability 
IDA Affecting Mission Accomplishment 


ce eee | 
Mission Accomplishment 
Compatibility Availability +> Interoperability 
Transportability Logistics supportability 


Reliability 
Maintainability 


Manpower supportability 


Human factors 


Safety 


Training requirements 
3/16/2007 Documentation 


Guidance Relating to DOD T&E 
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e Strategic Planning Guidance (SPG) 2006-2011: 

— “Development and fielding of joint force capabilities requires 
adequate, realistic test and evaluation in a joint operational 
context. To do this the Department will provide new testing 
capabilities and institutionalize the evaluation of joint system 
effectiveness as part of new Capability Based processes.” 


e Recommendation from SPG tasked Report, “Testing in Joint 
Environments (TIJE) Roadmap” : 


— “Establish acommon, TASK-based lanquage derived from 
UJTLs for use in Concept development, functional analyses, 
JCIDS capabilities, acquisition, T&E, training and experimentation 
and mandate its use in all JCIDS documents. 


What is Needed? How Might it Work? 
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a&» A Common, Task-based Language 
IDA Definitions of “Capability” 


¢ CJCSI 3170.01 11 May 2005, JCIDS 
The ability to achieve a desired effect under specified standards and 
conditions through combinations of means and ways to perform a set 
of Tasks. 


¢« Government of New Zealand, Pub 001, Corporate Statement of 
Intent(2004-2007), Appendix 5, Glossary SUITABILITY 


The appropriate combination of competent people, knowledge, 
money, technology, physical assets, systems and structures 


necessary to deliver a specified level of performance in pursuit of 
the organization’s objectives, now and/or in the future. 


¢ A Capability is a Task performed to Mission derived 
Conditions and Standards 


Capability-Based = 'Task-Based 
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PCO TWO Conceptual Models of the 
| Joint Operating Environment 


oo SCENARIO 


“SYSTEM-BASED” 
CONCEPTUAL MODEL 
Traditional Perspective 


C sysTENs 


Sees “Systems” 


: 


SUPPORTING 
TASKS Sees ALL Scenario Tasks 
{4 39 
ae TASK-BASED «Systems & Non-Kinetic 
*-PRODUCTS CONCEPTUAL MODEL 


«System Tasks 


“INFORMATION 


Capability-based Paradigm 


O ¢Cultural/Social Tasks 
¢Suitability 
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PON Joint Operational Environment 
1DA Two Conceptual Models 


SPG/DPS 
SCENARIO ~®, 
OPS A 
ANALYSIS / 
4 Pr ~ Needed 
[SYSTENS Sy Performance 
Available 
S ae (7st oon Test Conduct 
cenario-based 


METTI-T 


Mission Impact 
Assessment 


Task 


Metrics & Conditions 
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La Task Based Test Environments 
B! LVC Design Efficiency 


a ii ~~ 
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~~ 
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Scenario 


i foe CAPABILITY 
% DOTMLPF sep eat 
7 gee TEST 
» ° Performance ENVIRONMENT 
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CON Capability-based T&E 
IDA Test Design & Assessment 


DPS _ 
“Mission” MISSION 
_ Effectiveness o_—_aAar SCENARIO 
Missions .) oe = oe 
6 @ » TASKS 


TESTED SYSTEM 
ASA 


» 
Test Test 
Environment Performance 
‘%, Objectives 
*s, 
*, 7 SUPPORTING 
“e 7 TASKS 
*e av “OTHER SYSTEMS 
“SERVICES 
‘PRODUCTS 
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O = Non-Kinetic Tasks (SUITABILITY) 


2» Capability-based Suitability Testing 
IDA “Suitability” Components 


Supported Task 


MISSION <-—— 


Performance (demand) 


Documentation 
es) Available Performance 
Training A A Task done by 
\ “the System” 


Reliability = Maintenance “Demand” 


Maintenance “Available” 


Maintenance (a Supporting Task 


Parcaunél to the System) 
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POON Suitability 
| DA Affecting Mission Accomplishment 


SaaS SS 
Mission Accomplishment 
Compatibility Availability |*—_”| Interoperability 


Transportability 
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| Scenario 
Missions 
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Capability Assessment T&E Process 
Implementation Approach 
¢ “Establish a common, TASK-BASED language” 


¢ “Define Capabilities in common, measurable warfighting terms.” 


¢“Tdentify characteristics’ ability to accomplish or limit 
accomplishment of the mission” 


Task-based Mission Descriptions 


Distributed | 
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JCIDS 


COCOMS 


Common and Cross Cutting 
IDA Capability-based Conceptual Model 


“Establish a common, TASK-BASED Language, derived from 
UJTLs for use in Concept development, functional analyses, 
JCIDS capabilities, acquisition, T&E, training and 
experimentation and mandate its use in all JCIDS 
documents”; TIJE Road Map 


EXPERIMENTATION 


Common, Task-based, Mission Descriptions 


Conduct} Assess 


+ ft 
Distributed 


Item “Performance” 
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By Capability-based Suitability Testing 


pe eet SUMMARY 
e Task-based Conceptual Model of the Joint Operational Environment 
provides: 


— A comprehensive perspective on ALL activities in the Joint Operating 
Environment. 


— A useful paradigm for efficient & effective testing and assessment of 
system Suitability relative to Mission Accomplishment. 


¢ Task-based Conceptual Models are increasingly important to DOD 
decision making 


— Address today’s warfare processes more comprehensively than 
Systems-based paradigms. 


— Basis for an increasing number of DOD Assessment processes 


e System-based Conceptual Models are still needed to inform on MOP for 
Tasks at the Strategic, Operational and Tactical Level. 


— Anenduring role for traditional systems analysis and M&S 


3/16/2007 13 
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BACKUP SLIDES 


FoN A Common, Task-based Language 
LDA Task Performance Standards 


CJCSM 3500.04c, 1 July 2002 

e The standard for a joint task is set within the framework of the joint 
force commander's mission and in the context of the conditions, 
either most likely or worst case, that are linked to those missions. Thus, 


the standard(s) for a joint task can only be set when: 
— (1) the mission analysis is complete 
— (2) the conditions affecting the task have been identified and 


described, and 

— (3) measures and criteria have been selected that reflect the 
task contribution to mission accomplishment. 

This means that standards are tied to missions. That is, just because a 
joint task has a particular standard on one mission does not mean 
that the same standard will apply to other missions. 


To become expressions of “Capability” 
UJTLS require specification of scenario-based conditions and standards 
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CAPABILITY 
PON DOTMLPF, Scenarios, Missions, 
] DA Effects and Tasks 
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2 ask-based Adaptive Planning 


[DA Example 
— OPLANSCENARIO@ 
MISSIONS 

\ UJTL 
3 TASKS 
«Sustain 
: eTier 1JCA 
eTier 2 JCAS 
TPFDD | @®  .Tier 3 JMETLs 
e e@® UTCS @ @ © -Tier 4 Tasks 
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FORCES & RESOURCES 
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PON 
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*Related to Mission 
Accomplishment 


°A “Tasked Based” 
Expression of 
Performance 

*Useful to other 
Capability Based 
Communities 
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The Costs of Unsuitability and 
Benefits of Building in 
Reliability, Availability and 
Maintainability 


Dr. Ernest Seglie, 
Science Advisor, DOT&E 
Ernest.Seglie@osd.mil 


DoD IOT&E Results FY 2001 - 2003 


FY 2001 


Reliability, Maintainability, Availability 


Effective Reliability, Availability, Maintainability 


(RAM), Human Factors, BIT 


Integration with delivery platforms 


RAM, Safety, Human Factors 


RAM, excessive administrative and logistic 
FY 2003 
Effective 16% decrease in weapons release rate, 
reduction in accuracy of Mark 82 low drag 
weapons, 14% hit rate on moving targets 


Sea wolf Nuclear Attack Navy Effective Several requirement thresholds were not 
Submarine met but overall system effective and 


V-22 Osprey Navy 
Joint Direct Attack USAF 
Munitions (JDAM) 

M2A3 Bradley Fighting Army 
Vehicle 


1C 


FY 2002 


Joint Primary Aircraft 
Training System (JPATS) 


Multiple Rocket Launcher Army 
System (MLRS) 


B-1B Block E Mission 
Upgrade Program 


1D 
1D 
1C 
1D 
1C 
1c 
1D 
1D 


Cooperative Engagement Navy 
Capability (CEC) 


DoD IOT&E Results FY 2004, 2005 


FY 2004 
Evolved Sea sparrow Missile Navy Testing was not adequate to 
determine effectiveness. 
Stryker PaAmy | 4D | Effective | Suitable | SS 


Effective for short duration 
missions; not effective for all 
missions and profiles. 


Advanced SEAL Delivery System Navy 1D 
(ASDS) 
Not suitable due to RAM. 


Stryker Mortar Carrier-B (MC-B) RAM and safety concerns. 


FY 2005 
RAM; communications system less 


CH-47F Block | Army 1c Effective 
suitable than CH-47D; did not 
meet Information Exchange 
~ _ 


Requirements for Block I. 
ae 


ee MLRS | any 


RAM; needed more maintenance 
resources and spare parts; BIT 


Not effective against moderately 
hardened targets; mission 
planning time was excessive. 


en 


slice Mobility Attack Rocket System aaa Effective 
(HMARS) 
eiadie! pa af 


Air Force IOT&E Results 


FY 2002 


fssue [SEA Ralls 


Requirements Reasonableness RAM requirements not fully defined. BIT architecture 
Verification and subsystem reliability not designed into system. 
BIT system was a major requirement for the system. 


Program Planning | Allocation Program focused mainly on Band 1.5 and did not 


Sufficiency address newer SAM systems; inadequate processing 
capability. Systemic analysis was not performed; 
might have captured systems integration problems 
and identified root causes for inadequate processing. 


Acquisition Acceptability Program integrated Electronic Warfare (EW) systems 
Strategy with known reliability issues without performing a 
systemic analysis prior to design and integration. 


Technical Process | Requirements Development Program did not establish sound independent 
System Integration, Test, and technical review processes. Software assurance and 
Verification metrics not sufficiently established. Technical 
entrance and exit criteria not established for 
Developmental Test (DT) reviews and decisions. 


Air Force IOT&E Results 


FY 2002 


Joint Primary RAM; safety; human factors. 
Aircraft Training 
System (JPATS) 
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Requirements Reasonableness No ORD Thresholds for R&M; program measured 
Verification against objectives. 


Program Planning | Allocation Acquisition Reform - pilot program. Designed as 
Sufficiency COTS program. Multiple slips: evidence of a 
schedule-driven nature. Requirements not fully 
defined and understood. 


Acquisition Acceptability Simple COTS approach. “Militarization” not fully 
Strategy defined or understood. Multiple slips: evidence of 
schedule-driven nature. 


Technical Process | Requirements Development COTS mentality led to simplistic test approach (e.g., 
System Integration, Test, and FAA cert, Contractor Qual Test approach led to 
Verification insufficient DT). Multiple slips. Requirements not 
tracked/traced to a verification and test plan. 


Joint Direct Attack 
Munitions (JDAM) 


Air Force IOT&E Results 


FY 2001 


Excessive mission planning times 
(Navy); system reliability; B-52 
load times; container deficiencies 
(stacking, carrier ops). 


fssue [SEA [Rane 


Requirements 


Acquisition 
Strategy 


Technical Process 


Reducibility and 
Production 
Planning 


Reasonableness; Design Synthesis 
Verification 


Acceptability 


System Integration, Test, and 
Verification 


Quality Control (Plant Layout) 


B-52 load times not reflective of new complexity. 
Navy carrier operability (ruggedness) not adequately 
captured/defined. Significant focus on capability 
(accuracy). Reliability relied heavily on “warranty.” 


Acquisition Reform -— pilot program. Small program 
office. Capability-based contracting strategy; 
significant SE contracted as result. Significant focus 
on capability (accuracy). Reliability relied heavily on 
“warranty.” 


Unrealistic load times; test team load crew 
experience, training; test team mission planning 
experienceltraining. 


Storage reliability. Significant failures related to 
minor quality control errors (i.e., missing sealant, kit 
packed with wrong covers, etc.). 


Air Force IOT&E Results 


FY 2003 


B-1B Block E Msn 16% decrease in weapons release 

Upgrade Program rate; reduction in accuracy of Mark 
82 low drag weapons; 14% hit rate 
on moving targets. 


a 


Requirements Reasonableness Validity of effectiveness measures, based on 
Verification comparison with prior block (not as complex; 
different release mechanism; different weapons mix; 
key requirement met: weapons flexibility). 


Acquisition Acceptability; Sufficiency Software conversion oversimplified. Significant 

Strategy program growth. “Program clarity” - funded program 
did not address numerous “known issues”; resulted 
in re-identification of numerous issues (situational 
awareness, controls and displays, reliability). 


Technical Process | System Integration, Test, and T&E measures not well founded in ORD/CDD. 
Verification 


Air Force IOT&E Results 


FY 2003 
1D Effective RAM; needed more maintenance 
resources and spare parts; BIT. 


fssue [SEA [Ralls 


Requirements Reasonableness RAM requirements not fully defined for IOT&E but for 
Verification a mature aircraft at 100K flight hours. RAM and BIT 
requirements not tracked/traced to a verification or 
test plan. 


Acquisition Acceptability; Sufficiency Program did not recognize or fully fund RAM 

Strategy requirements and software development, especially 
the maintenance software portion. Labs were 
insufficiently supported with hardware-in-the-loop. 


Technical Process | System Integration, Test, and Program did not establish entrance/exit criteria for 
Verification software development, verification, validation, and 
test. Software not adequately tested and fixed in the 
lab prior to flight test. Mission technical issues 
overshadow RAM issues and RAM resources diverted 
to technical mission issues. Program did not have a 
sound risk assessment program. 


Additional Costs When a System 
is Judged Unsuitable (1) 


¢ Some programs extended their SDD and added 
resources to redesign, reengineer and to retest till 
they became suitable 


— \-22 extended its SDD by five years and spent ~$1B to 
resolve its suitability issues. (It had a catastrophic failure in 
2000) 


— C-17 Is likely to be another interesting case. 

¢ When failure in OT&E delays the full production and 
the fielding of a new system, it may require extra cost 
to operate and support, and in some cases, even 
service Life Extension Program (SLEP) on legacy 
systems. 


Additional Costs When a System 
is Judged Unsuitable (2) 


¢ Some programs were granted FRP and delayed RAM remedial 
actions as Block Upgrades 


— Approach requires additional cost for RDT&E and retrofit. It is also 
more expensive to maintain and support several different 
configurations than one. 


— It turns to a spiral development approach. 
— Identify related RAM development and retrofit costs 


— Estimate additional operating and support costs for extra 
configurations 


¢ Some programs are fielded with known RAM shortcomings 


— Extra costs for repair and maintenance or contractor logistic support 
when fielded at insufficient RAM level 


— Possible cost to procure and operate additional units to compensate 
for low availability to meet desired sortie rates or ton-mile capacity 


LCC Distribution 
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Life Cycle Management 


Pre-Systems 
Acquisition Systems Acquisition Sustainment 


Four Causes 


No requirements 

Lack of incentives 
Attention elsewhere 

Poor Systems Engineering 


JROC Memo: 17 Aug 2006 


“MATERIEL AVAILABILITY” KPP for all MDAPs and Select ACAT Il 
and Ill 


(KSAs): 
A. Materiel Reliability KSA 
B. Ownership Costs KSA 


JROC Approved* Mandatory 
Sustainment KPP and KSAs 


Single KPP: 
¢ Materiel Availability (= 


Mandatory KSAs: Total Operating Hours 
. . oe _ 10ta!i Uperating hours 
Materiel Reliability (MTBF)(= 7 oy bor of Failures) 
¢ Ownership Cost (O&S costs associated w/materiel readiness) 


Ownership Cost provides balance; solutions cannot 
be availability and reliability “at any cost.” 


Number of End Items Operational 
Total Population of End Items ) 


*JROC Approval Letter JROCM 161-06 Signed 17 Aug 06; 


Revised CJCS 3170 will put into Policy 


Return on Investment 


Estimate O&S and Initial Spares 


of Different F-22 MTBMs 
(Constant 2006 $B) 


Reliability Level at MTBM in O&S & Initial Life Cycle Cost 
Maturity Hours (1) Spares (2) Difference (3) 
FOT&E Actual (1a) 0.65 $ 42B $ 7B 


IOT&E Actual with 
Historical Growth (1b) 0.83 $ 40B $ 5B 


Air Force Program 
Reliability Projection (1c) 1.50 $ 35B 


(1) Mean Time between Maintenance. F-22 ORD established MTBM threshold at 3 hours. 
(1a) MTBM of 0.65 hours achieved in Follow-on Operational Test and Evaluation (FOT&E). 
(1b) IOT&E MTBM score 0.45 hours. F-22 will achieve MTBM of 0.825 hours at maturity (100,000 FH), if its reliability 
growth rate is similar to the historical rates of existing fighter aircraft programs. 
(1c) Air Force Program Office projects F-22 to achieve 1.5 hours MTBM at maturity. 
(2) O&S cost for 148 Primary Aerospace vehicle Authorization (PAA), 336 flying hours per aircraft per year for 24 years. 


Initial spares requirement for 182 Total Active Inventory (TAD, computed at $120M recurring flyaway cost each. 
(3) Baseline assumes the Air Force projected 1.5 hours MTBM at maturity. At the F-22 ORD MTBM threshold of 3 hours, 


Return of R&M Investment 
(Present Value 2006 $B) 


Reliability Level at MTBM in O&S & Initial RDT&E& Savings to 
Maturity Hours (1) Spares (2) Retrofit (3) Investment Ratio 
FOT&E Actual 0.65 $ 30B 
Air F Pp 
eee ee 1.50 $ 25B 
Reliability Projection 
Potential Savings (4) $ 5B 
Budgeted Investment $ 0.7B 
Potential Return of 734 
Investment } 


(1) Mean Time between Maintenance. F-22 ORD established MTBM threshold at 3 hours. 

F-22 Follow-on Operational Test and Evaluation (FOT&E) MTBM score 0.65 hours. 

Air Force Program Office projects F-22 to achieve 1.5 hours MTBM at maturity. 
(2) O&S cost for 148 Primary Aerospace vehicle Authorization (PAA), 336 flying hours per aircraft per years for 24 years. 

Initial spares requirement for 182 Total Active Inventory (TAI), costed at $120M recurring flyaway cost per aircraft. 

(3) President Budget Submission (February 2005 and February 2006): 

F-22 Reliability and Maintainability Maturation Program (RAMMP). 

F119 engine Component Improvement Program (CIP). 

R&M retrofits: air vehicle RAMMP modification and F119 engine CIP modification 


(4) Saving will be substantially lower if F-22 does not achieve MTBM of 1.5 hours at maturity. 


R&M Investment and Savings 
(PV 2006 $B) 


a 
Saving to Investment Ratio | ~--7~ 
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27. $1°3B R&M Investment: $0.7B 


0.8 0.9 1.0 Il 1.2 1.3 


MTBM (hour) at Maturity 


1.4 


Defense System Life Cycles 


| | | | 
| 


| the F-15 > 51 yrs 
CH-47. . | | | | | | 71 yrs 
M-113 eS 58 yrs 

ja, UH-1 


a. FKC-135 
=3*— AIM- ! 
— C-130 


2.5 Ton Truck ! 


1940 1950 1960 1970 1980 1990 2000 2010 2020 2030 2040 


SOURCE: John F. Phillips DUSD (L) 


Myths about Building-in 
Reliability, Availability and 
Maintainability 


Myth 1: Building-in Reliability costs 
money. 


HH-60H and MH-60S Reliability and 
Cost Comparison 


Component Component 


CPU133/A DIGITAL 
COMPUTER 


AUXILIARY POWER 

SYSTEMS ACFT POWER UNIT 
SECT'S 2/3/4 DRIVE SECTIONS 2/3/4 DRIVE 
SHAFT ASSY SHAFT ASSY 


CP1820/ASN150 NAV CP-2428/A DIGITAL 
COMPUTER DATA COMPUTER 
STABILATOR AMPLIFIER 
AMPLIFIER INSTALL INSTALLATION 
MLG DRAG BEAM-AXLE 
BEAM/AXLE ASSY ASSEMBLY 
FLOOR ASSEMBLY AIRCRAFT FLOOR 


T1360( VALQ144(V) LIGHT, INFRARED 
TRANSMITTER TRANSMITTER 


* 0 failures observed in one year 


When to Invest? 


100 


10 


SCAMP 
JSTARS CGS 


The Ratio of Developmental Test Reliability to Operational Test Reliability 


DT reports results that are 100 times “better” than OT 


DT reports results that are 10 times “better” than OT 


Red bars are systems where DT concluded reliability MET, but OT concluded NO MET. 
Black bars are systems where DT and OT concluded NOT MET 


Green bars are systems where DT and OT concluded MET 
Brown bars are systems where DT concluded NOT MET, but OT concluded MET 
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Return on Investmer 


¢ Seriesl 


Investment Cost ($ 


Reliability (MRBF) 


Raesi n 


Acaulereet 


S Redesign 


Test Rounds ~ Cost 
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Reliability (MRBF) 


Rounds ~ Cost 


Inventory 3 N needed at.3 requirements reliability 


Inventory 2 N needed at.5 requirements reliability 


Inventory N needed at requirements reliability 


Rrequirement 


Reliability (MRBF) 


Inventory 3 N needed 
at 3 requirements reliability 


Rounds ~ Cost 


Inventory 2 N needed 
at 5 requirements reliability 


Savings from Earl 
Investment -> 


Savings from Late 
Investment -> 


Inventory N needed 
at requirements reliability 


Late Investment -> 


Early Investment <7 


Kauean 


Reliability (MRBF) 


Summary 


¢ How to Address Problems 
¢ Size of ROIs? 
¢ When to invest? 


THE END 


Rounds ~ Cost 


Inventory 3 N needed at .3 requirements reliability 


Inventory 2 N needed at .5 requirements reliability 


Inventory N needed at requirements reliability 


Rrequirement 


Reliability (MRBF) 


SDD Investment for 
Operational Suitability 


¢ Compare and contrast SDD/EMD approaches of 
cases with different emphasis on RAM (F/A-18 
C/D vs. E/F; F-15 vs. F-16) 
— Design and Engineering 
— Schedule, costs, test articles 

¢ Compare and contrast distinct DT approaches 
taken in cases and resulting suitability rating 
(JSOW, JASSM) 
— Standards and specifications 
— Schedule, cost, test articles 


¢ Contractors’ proposed fixes and associated costs 
for RAM issues 


reduce total ownership costs (R-TOC) 


grew out of (PEO/SYSCOM) Commanders' conferences, the Defense Science Board, and 
others. The R-TOC program was established in response to longstanding concerns about 
the adverse impact of defense budgetary and operational trends on force structure and 
readiness. Rising operations and support (O&S) costs can consume higher portions of 
defense budget and leave even less available for modernization. 


The purpose of the R-TOC program is to achieve readiness improvements in weapon 
systems by improving the reliability of the systems or the efficiency of the processes used 
to support them. New Technologies and management practices may provide significant 
opportunities to improve readiness and reduce ownership costs. In recent years, world- 
class suppliers have achieved cost reductions while making major improvements in 
customer support. Some DoD programs have achieved similar successes in adopting 
private sector improvements in logistics and supply chain management. 


Weapon Systems 
CH-47 
Multiple 
UH-60 
AH-64 
CH-47 & AH-64 
H-47/H-64 
Ground Support 
Patriot Missle 
B-1B 
A-10 
F-16 
C-130 
B-52 
C-5A/F-15 
F-15 
F-15/Multiple 
F-15 & F-16 
KC-135 
F100/F-119 


Minuteman 


Multiple (Fuel) 


F/A-18 

Multiple (Navy) 
EA-6B 

H-60 

AV-8B 

T-64 

F404 
AH-1/H-46 
C-130/P-3 
CH-46 

E2-C-2 

EA-6B, E-2/C-2 
F/A-18 & E2/C2 
H-1 

HMMWV 

T-58 

T700 

UH-1 


Assessing Cost and 


¢ Models to link Mean Time Between Maintenance 
and other R&M metrics to requirements for 
— Maintenance manpower 
— Sustaining spares 
— Initial spares 
¢ F-22 example 


— Compare the O&S cost and initial soares requirement for a 
range of reliability assumptions, following established 
analytical approaches for other F-22 Studies. 

— Tabulate the trade-off between budgeted F-22 R&M 
investment (RDT&E and retrofit) and potential O&S and 
initial spares savings. 


RAM DESIGN: 
Reliability Design Technique 


* Mission profile * Allocation 


mm aloes analyele * Parts selection 


* Worst case analysis 


* Derating criteria 


* FMECA 
* Simplification 


* Sneak circuit 


* Design reviews 


Proposed Life Cycle Sustainment 
Outcome Metrics 


¢ Materiel Availability (KPP*) 
— A Key Data Element Used In Maintenance 
And Logistics Planning 
¢ Materiel Reliability (KSA*) 
— Provides A Measure Of How Often The System Fails/Requires Maintenance 
— Another Key Data Element In Forecasting Maintenance/Logistics Needs 
¢ Ownership Cost (KSA*) 
— Focused On The Sustainment Aspects Of The System 
— An Essential Metric For Sustainment Planning And Execution 
— Useful For Trend Analyses — Supports Design Improvements/Modifications 
¢ Mean Downtime 
— A Measure Of How Long A System Will Be Unavailable After A Failure 
— Another Key Piece Used In The Maintenance/Logistics Planning Process 


¢ Other Sustainment Outcome Metrics May Be Critical To Specific 
Systems, And Should Be Added As Appropriate 


* Sustainment KPP & KSAs Included In Revised Draft CJCSM 3170 


These 4 Life Cycle Sustainment Outcome Metrics Are Universal 


Across All Programs And Are Essential To Effective Sustainment Planning 


otryker Findings & Observations 


¢ Warfighters very satisfied with Stryker 
performance in-theatre 


¢ Brigade Commanders extremely happy with ICLS 

¢ High Operational Readiness Rates, but ORR Is 
prioritized over support costs 

¢ Op Temp in-theatre far exceeds planned usage 
rates (X10, X15, X30 7?) 

¢ Operational Environment much different than 
expected 

¢ Combat configurations add excessive weight to 
vehicle (affecting reliability and performance) 

¢ Army did not buy Tech Data Pkg - “Prohibitively 
expensive” ... risk to government 


Findings & Observations 


Operational Readiness Rate not necessarily 
consistent with traditional A, 
— RAM issues can be masked by ORR 


Mission Completion vs. Subsystem Failure 
— Possibly leads to overestimating system reliability due to non- 
reporting on individual subsystem (component) failures 
— Multi-mission vehicle — with subsystem failures, system can still 
perform alternate missions 
Reporting Criteria Issue: 
— ORR vs. MTBF of individual subsystems 


Reliability Issues 


¢ RAM requirement poorly defined in ORD 


— 4.3.1.3. The Stryker (vehicle only, excluding GFE 
components/systems) will have a reliability of 1000 mean miles 
between critical failure (i.e., system aborts). 


¢ Reliability issues and cost drivers found 
during DT/OT correlate well with in-theatre 
experience 


Operational Environment 


Field usage much harsher than planned 
— e.g., higher tire pressure, roads, curbs, weight (armor, sandbags) 
OMS/MP says 80% XCountry, 20% Primary Roads 


— in-theater mission just the opposite . . . most missions in urban 
environment (police action) on paved roads 


OpTempo very high (>10X) 
— High OpTempo may improve reliability numbers, but beats up equipment 
— With low usage, seals can dry up, humidity can build up in electrical 
components 
Changes in mission & configuration are putting 
excess stress on vehicle: armor/sandbags, over 
inflated tires, going over curbs 
— replacing 9 tires/day (>3200 tires/yr) 
wheel spindles developing fatigue cracks 
drive shafts breaking 
prescribed tire pressure is 80 PSI, however, with slat 
armor/sandbags — must maintain >95 PSI 
95 PSI is a logistics burden on operators 
¢ Must be maintained by the soldier (CTIS system can’t do it) 


¢ Soldiers must check tire pressure more than 3 times per day to 
maintain 95 PSI 


otryker Fleet Readiness 


Number Fielded 


ORR vs Strykers Fielded 
As Of: 20 Feb 2007 


mn 
Ml 


SS 


Ga Strykers Fielded —e— ORR 


[o) ° fo) 
ORR Percentage 


Operational Readiness Rate (ORR) 


¢ PBL Contractual Requirement: ORR > 90% 
— Does not include GFE (base vehicle configuration only) 


¢ Consistently above requirement 
— Current ORR 97% (20 Feb 07) 
¢ Cost-plus-fixed-fee contract motivates GDLS to meet 
ORR.... 
— However, contract does not incentivise controlling costs... 
risk to government 


— Example - to repair cracked hyd res in power pack, whole 
power pack is replaced in field 


Cost Estimate Variation 


CPM estimate - $17.19 (GAO 04-925, including labor, parts & repair) 

CPM estimate - $18.78 (Stryker R-TOC Brief) 

CPM estimate - $18.23 (based on M113 methodology w/Stryker adjustments) 
CPM estimate - $14.53 (based on initial 4 month deployment data) 

CPM estimate - $12.72 (GDLS estimate, labor, parts & repair) 


We need to understand the basis for these estimates more thoroughly 
(assumptions, models, configurations, limitations... ) 


Tactical Considerations 


Slat Armor design (additional 5000 Ib) is effective for many RPG 
threats, but negatively impacts circumference, weight and 
performance of Stryker 
— Causes multiple problems for safe and effective operation 
¢ Slat armor on rear ramp too heavy - greatly strains lifting equipment 
— Occasionally, crews must assist raising/lowering ramp 
¢ Bolts on rear ramp break off frequently with normal use 


¢ Slat armor bends with continued ops ...can cover escape hatches and 
block rear troop door in ramp 


¢ Interferes with driver’s vision 
¢ Slat armor difficult to see at night... Safety hazard in urban environment 
¢ Slat Armor prohibits normal use of exterior storage racks 
— Significantly impacts handling/performance in wet conditions 
¢ Excessive strain on engine, drive shafts, differentials 
— Impairs off-road ops 
Though not designed primarily for the urban fight (MOUT), 
Stryker is well-suited for it 
— Unlike M-1, Stryker is “Ghostly” quiet... tactical advantage 
Stryker overall OIF performance significantly better than 
HUMVEE, BRADLEY or M-1 in this environment 


Other Findings ... cont. 


Stryker initial deployment/fielding was extremely accelerated to 
meet urgent combat need 

— Result was that Army was doing these things concurrently: 

* Testing 

¢ Producing 

¢ Fielding 

¢ Conducting combat operations 
Immature Maintenance Procedures (many procedures have not 
been validated in IETMs) lead to: 

— “Tribal System Maintenance” from experienced crews (that new book 
isn’t any good........... this is the way it worked on the M113, so 
do it like this) 

With Kr support to maintain vehicles, soldier crews develop 
“rental car mentality” .. . 

— Overly dependent on Kr.... lack of ownership mentality 

— Sometimes they forget the basics (oil check) 

— One vehicle lost because pre-mission checks were ignored 

¢ maybe lack feeling of ownership, maybe overly depend on Kr 


Typical Suitability Requirements 


Logistic Support 

— Deployability of Forces and Equipment 
Training 

— Ability to Train Maintenance Personnel 
Ability to Launch Aircraft 

— Sortie Generation Rates (Peacetime and Wartime) 
Reliability and Maintainability (R&M) 

— Break Rate, Fix Rate 

— Mean Time Between Maintenance (MTBM) 

— Mean Time Between Removals (MTBR), Mean Down 
Time 

Mission Capable Rate and Weapon System Reliabilities 


Elements Affected by 
R&M 


¢ Poor R&M leads to more frequent maintenance/repair 
and longer time to repair/replace. 
— More initial spares => higher procurement cost 
— More repair parts => higher operating and support (O&S) cost 
— More maintenance manpower 
= > higher O&S cost 
¢ Tocorrect the shortcoming in R&M after full rate 
production requires: 
— Operational System Development to design and test fixes 
= > additional RDT&E cost 
— Retrofit existing production units 
= > additional modernization cost 
¢ Poor R&M lowers sortie generation rate, could 
necessitate additional procurement 


C-1/ MTBM (Cumulative) 


MTBM 


0.1 
1000 


a=0.35 


10000 
FH 


100000 


¢ MTBM 
—— MTBM (Duane) 


F-22A Mean Time Between 
Maintenance (MTBM) 


10.00 
1.00 ‘ 
0.10 
@ a= 0.44 
0.01 
10 100 1000 10000 


OT&E Flight Hours 


100000 


@ MTIBM(Cum) 
——— MTBM (Cum fit) 
MTBM (Inst fit) 
e@ MITBM (Inst) 


IMEASURE 


Manpower 


Maintenance Manpower 
vs. Calibration Factor for Reliability 


250 


F16 Data 


Sd 


A F22 Data (13.5 hr 
repair time) 
— — Log. (F16 Data) 


ei y = -43.7Ln(x) + 131.8 


Log. (F22 Data (13.5 
hr repair time) ) 


150 


100 


50 i 2 
= -47.5Ln(x) + 79.6 
R? = 0.99 


0 0.5 1 1.5 2 2.5 
Calibration Factor 


Estimate O&S and Initial Soares 


of Different F-22 MTBMs 
(Present Value 2006 $B) 


Reliability Level at MTBM in O&S & Initial Life Cycle Cost 
Maturity Hours (1) Spares (2 Difference (3) 
FOT&E Actual (1a) 0.65 $ 30B $ 5B 
IOT&E Actual with 
Historical Growth (1b) 0.83 $ 29B $ 4B 


Air Force Program 
Reliability Projection (1c) 1.50 $ 25B 


(1) Mean Time between Maintenance. F-22 ORD established MTBM threshold at 3 hours. 
(1a) MTBM of 0.65 hours achieved in Follow-on Operational Test and Evaluation (FOT&E). 
(1b) IOT&E MTBM score 0.45 hours. F-22 will achieve MTBM of 0.825 hours at maturity (100,000 FH), if its reliability 
growth rate is similar to the historical rates of existing fighter aircraft programs. 
(1c) Air Force Program Office projects F-22 to achieve 1.5 hours MTBM at maturity. 
(2) O&S cost for 148 Primary Aerospace vehicle Authorization (PAA), 336 flying hours per aircraft per year for 24 years. 
Initial spares requirement for 182 Total Active Inventory (TAI), computed at $120M recurring flyawy cost each. 
(3) Baseline assumes the Air Force projected 1.5 hours MTBM at maturity. At the F-22 ORD MTBM threshold of 3 hours, 


the estimated life cycle cost would be $3B lower than the baseline in 2006 present value. 
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ARMY TEST AND EVALUATION COMMAND 


Xs kes 
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¢ Plan, conduct, and report the results of tests, 
simulations, experiments, and evaluations to 
Acquisition decision makers in order to ensure 
our Army’s Warfighters have the right 
capabilities for success across the entire 
spectrum of operations. 


¢ Conduct rapid testing in direct support of the 
GWOT warfighter in order to provide capabilities 
and limitations of untested weapon systems 
issued directly to Soldiers conducting combat 
operations (Iraq/Afghanistan). 


NA SE ee oe. Oe 


is 


¢ Standards are the same 
- GWOT 


- Normal Acquisition 


¢ Test Strategy tailored to meet 
Acquisition Strategy 


¢ Must answer 2 Questions 


- “Does it work...How do I know?” 
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Virtual/Features 


4 Individual & Collective 
Virtual Range Protection & Survivability 


Evaluation 
Report 


acs cATS AE Test 


AWK LEK AD SDE TF yd 


Streamlined 
¢ T&E Concept 
¢ Testing 


Fixes in 


hours and 
days! 


pri 
Test-Fix-Test Confirmation 


Capabilities & 


Limitations 
Report (CLR) 


GWOT DTI/OT 
Less Than 
90 Days 


GWOT C&L 
Less Than 
14 Days 


DTIOT - 44 Days 


C&L Report - 12 Days 


Armor Coupon Shots - 
48-hr Response 
to Theater 


SLE RAVES 


i SE Sak 
¢ Tailor each weapon system T&E concept 
¢ Full Spectrum vs. OIF/OEF | 
¢ Environmental 
¢ Durability: Combat Opns today vs. Full Life Cycle Reliability 


¢ Interoperability 

¢ TTP/CONOPS 

Training (Leader, Individual, Collective, Sustainment) 
Logistics Concept 

¢ Not all systems fit the Rapid Testing “Model” 


SAW LEK ALD SED Tt 


MISSION: 


Provide Essential Info 
to Sr. Army Leadership 


Test & Collect data on 
selected Warfighting 
systems 


Multi-Unit Liaison with 
MNC-I, CFLCC, CJTF- 
76, and CENTCOM HQs 


Obtain direct feedback 
from Soldiers 
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¢ ATEC Fully Engaged in GWOT 

¢ Not all systems fit the Rapid Testing “Model” 
¢ Right Size the Test 

¢ Mitigate the Risk 

¢ Follow the System thru GWOT & Beyond 


¢ “Does It Work...How Do I Know?” 
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ACADEMY 
ISON 


Simplify, Perfect, Innovate 


Wifstetsiel asvanlsjalmrslale Mm B=) (sleil(e)ame)r-laletsigels 
from 
The Theory of Inventive Problem Solving 


Michael S. Slocum, Ph.D., T.Sc., M.B.B. 
(@lal(siimfalarencclucelame@viilasys 
Air Academy Associates 
(Oto) (e) ¢-Tele me) 6) dl ale ise OOo 10}o 740) 
aaksilelere lanl Gers les \er-lemere) an 


TRIZ: The Theory of Inventive Problem Solving 


= Systematic Problem Solving Methodology 
¢ Empirically Derived Techniques 


— ldeality 
— Technical Contradiction Theory 
— Levels of Innovation 
— Maturity Mapping 
— Technology Forecasting 
¢ Heuristics 


— Psychological Bias 

Ideal Final Result 

Physical Contradiction Theory 

System Approach 

— ARIZ 

Are — Substance-Field Modeling 

AGRA TES — 76 Standard Solutions and algorithm 


Simplify, Perfect, 


Innovate 


©2007 Air Academy Associates, LLC. Do Not Reproduce. 


SFM: Substance-Field Modeling 


=" Modeling Technique 


¢ Describes the Interactions within a System 
Triad: 
— Field, F 
— Substance 1, S, 
— Substance 2, S, 


ry interaction 


interaction 


Simplify, Perfect, 
Innovate 


©2007 Air Academy Associates, LLC. Do Not Reproduce. 


The Fields, Fy 


= Mechanical, F,,. 

= Electrical, F, 

= Thermal, F,,, 

= Chemical, Fo, 

= Magnetic, F, 

= Electromagnetic, F,,., field 


= Nuclear, F,, e 
= Compound Fields: | | | | 
Interactio Interaction 
° E-Me vA 
¢ Th-Ch Q 


* E-Ch, etc. Interaction 


Simplify, Perfect, 


Innovate 


©2007 Air Academy Associates, LLC. Do Not Reproduce. 


Simplify, Perfect, 
Innovate 


The Substances: S, and S, 


= S, is Known as the workpiece 

= S, is the tool 

= The tool is applied to the workpiece and the 
interaction is described by a field and the 
resultant interactions 

= Energy flows from the tool 


to the workpiece i 


interactio interaction 


workpiece 0... interaction tool 


©2007 Air Academy Associates, LLC. Do Not Reproduce. 


The Interactions 


= Sufficient: ———> 


=Insufficient 77" ""™ 


= Harmful: qQwAAA 


ry interaction 


interaction 


Simplify, Perfect, 
Innovate 


©2007 Air Academy Associates, LLC. Do Not Reproduce. 


Are 


Simplify, Perfect, 
Innovate 


Modeling Examples 


#S1: wall 
#S2: person 
#F: painting 


2S1: wall 
#S2: paint 
=F: adhesion 


2S1: wall 


#S2: brush 
#F: application 


©2007 Air Academy Associates, LLC. Do Not Reproduce. 


#S1: paint 
#S2: brush 
#F: application 


#S1: brush 
#S2: person 


#F: moving application 


Incomplete, Insufficient, or Systems with Harm 


= The minimum functioning system model involves 
a complete triad: F,, S,, S,, and each interaction 
(3 minimum) 
¢ A system that is missing any of those elements is 
incomplete 
e¢ An incomplete should be 


completed e 
ry interaction 


interaction 


Simplify, Perfect, 


Innovate 


©2007 Air Academy Associates, LLC. Do Not Reproduce. 


The Incomplete Model 


One or two elements of 
the minimum technical 
system are missing 


Missing F, 


Missing S, and F, Missing S, and F, 


Simplify, Perfect, 


Innovate 
©2007 Air Academy Associates, LLC. Do Not Reproduce. | 


cn 


Simplify, Perfect, 


Innovate 


Standard Solution 


Add the missing element 

of the model: F,,, S,, or S, 
Explore each field: Me, E, 
Th, Ch, M, EM, or N 


©2007 Air Academy Associates, LLC. Do Not Reproduce. 


Simplify, Perfect, 


Innovate 


Incomplete, Insufficient, or Systems with Harm 


= The minimum functioning system model involves 
a complete triad: F,, S,, S,, and each interaction 
(3 minimum) 
e A system that contains at least one insufficient 
interaction is an insufficient system 


e An insufficient system should be made 
sufficient by adding the necessary 
elements to resolve the 


i. interactio interaction 
insufficiency . 


interaction 


©2007 Air Academy Associates, LLC. Do Not Reproduce. 


Insufficient Model 


An insufficient model 
reflects a situation where 
all three elements are in 
place, but the useful field 

(F) is not sufficient @ 


Simplify, Perfect, 
Innovate 


©2007 Air Academy Associates, LLC. Do Not Reproduce. 


Standard Solution 


Insert a substance (S3) 
which is a modification of 
S1 or S2 or both that 
causes the interaction to 


become sufficient @ 
A la 
9--8 


Are 


Simplify, Perfect, 
Innovate 


©2007 Air Academy Associates, LLC. Do Not Reproduce. 


Incomplete, Insufficient, or Systems with Harm 


= The minimum functioning system model involves 
a complete triad: F,, S,, S,, and each interaction 
(3 minimum) 
¢ A system that contains at least one harmful interaction 
is a system with harm 
e Asystem with harm should be made 


ry interaction 


interaction 


to have no harm 


Simplify, Perfect, 


Innovate 
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System with Harm 


All three elements are In 
place, but the interaction 
between S1 and S2 Is 
harmful or undesired or 
F, is harmful 


Simplify, Perfect, 
Innovate 
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Standard Solution 


Replace F, (or F, and S.,) 
with another set: F, or (F, 


and S.) 
Explore each field: Me, E, Fy 


Th, Ch, M, EM, or N / \ “9.8 


me , 


Standard Solution 


Add another field (F.) to 
intensify the desired effect 
Explore each field: Me, E, 
Th, Ch, M, EM, or N 


P» 


Age 


Simplify, Perfect, 
Innovate 
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Are 


Simplify, Perfect, 
Innovate 


Standard Solution 


Insert a substance (S.) 
and add another field (F.). 
S, and F, enhance the 
desired effect 


Explore each field: ’ } 
Me, Th, Ch, E, M, EM, /\ /\/ 
or N 
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Standards 


= There is a body of standard solutions 
called the 76 Standard Solutions 


ry interaction 


interaction 


Simplify, Perfect, 
Innovate 
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. The 76 Standard Solutions: 5 Classes 


> CLASS 1. COMPOSITION AND DECOMPOSITION OF SFMS 
GROUP 1-1: SYNTHESIS OF A SFM 
GROUP 1-2: DECOMPOSITION OF SFMS 
CLASS 2. EVOLUTION OF SFMS 
ge GROUP 2-1: TRANSITION TO COMPLEX SFMS 
GROUP 2-2: EVOLUTION OF SFM 
GROUP 2-3: EVOLUTION BY COORINATING RHYTHMS 
GROUP 2-4: F 
CLASS 3. TRANSITIONS T 
GROUP 3-1: T 
GROUP 3-2: TRANSITION TO MICROLEVEL 
CLASS 4. MEASUREMENT AND DETECTION STANDARDS 
GROUP 4-1: INSTEAD OF MEASUREMENT AND DETECTION - SYSTEM 
CHANGE TO SOMETHING ELSE 
GROUP 4-2: SYNTHESIS OF A MEASUREMENT SYSTEM 
GROUP 4-3: ENHANCEMENT OF MEASUREMENT SYSTEMS 
GROUP 4-4: TRANSITION TO FERROMAGNETIC MEASUREMENT 
STS 
GROUP 4-5: EVOLUTION OF MEASUREMENT SYSTEMS 
CLASS 5. SPECIAL RULES OF APPLICATION 
GROUP 5-1: SUBSTANCE INTRODUCTION 
GROUP 5-2: INTRODUCTION OF FIELDS 
GROUP 5-3: USE OF PHASE TRANSITIONS 
GROUP 5-4: PHYSICAL EFFECTS USE 
GROUP 5-5: SUBSTANCE PARTICLES OBTAINING 


How Does it All Work? 


Identify the measurement and/or detection 
problem(s) in the system 
Create a SFM of the problem area(s) 


Analyze the SFMs for any deficiencies 
— Incomplete 
— Insufficient 
— Contains harm 


Utilize the SFM Algorithm to identify the 
appropriate solution standard(s) 

Apply standard solutions to create generic 
approaches for problem resolution 

Convert generic approach to specific solution 
using analogic thought 


< 76 Standards Algorithm 


| 


i 2. Transform model to Su-Field form 
oe 4.1 Harmful link No 
(o) 
4. Su-Field Complete present? 


i 
ne 
\ 
1 
\ 
‘ 


: Yes 
\ NG 4.1.1 Introduction of substances 
\ 5. Harmful link present? Andinalde alowed 
\ 
\ 
Use standards 1.1.1 to 1.1.6 or 
Yes 
4.2 Introduction of substances 
Use standards 1.2.1, 1.2.2, 1.2.4, 1.2.5 and fields allowed? 
5 


] 
a 


ee 
=< : 1. Build a model of the problem 
—— Yes 


‘ Yes 


5.1 Introduction of substances NG 


and fields allowed? 


Air 
Academy 


Yes 
Associates Use standards from groups 5.1, 5.2, and Use standards et eee elle oy 
5 2s 


Copyright 
2007 


6. Are there magnetic 


' ; Use standards 2.4.2, 2.4.3, 
substances in Su-Field? 17. Is Su-M_field dynamic? ae 
| Tee uo 78 a ee 
7, Introduction of M_ Field as SUM anee elements Use standards 2.4.9 
structure coordinated? 


allowed? 


8. Complex Su-Field 
formation allowable? 


Yes 


Use standards 2.4.10, : 
groups 5.3, 5.4 


19. Are Su-M_field elements 
dynamics coordinated? 


- Ye 
; No 
9. Su-Field can be 
replaced? Use standard 2.2.1 


Use standards 2.4.1 or 


Use standards 2.4.5 or 


10. Is the system Use standards 2.2.2, 2.2.3, 
\ dynamic? 2.2.4 
x 


Use standards 2.2.5, 2.2.6, or 
and groups 5.3, 5.4 


11. Is Su-Field elements 
structure coordinated? 


\ 
\ 
\ 


12. Is Su-Field elements 
\ dynamics coordinated? 


Use standards 2.3.1, 2.3.2, 
Ps Sy (OVE 


. Introduction of ferromag: 
additives in environment 


13. Substitution of Su-field 
elements by ferromag. subst. and 


Air M_Field allowed? allowed? 
Academy Apply standards from 
Associates Yes group 3 in the following 
Piao: 14. Introduction of ferromag. 


OURO StL, Sill, Shelly 
Shells}, Shells) 


16. Is use of electrical fields 


additives in available subst. 
and/or currents allowed? 


allowed? 


No 


i Conclusions 


) ¢ Problem solving may often include the need 
for the collection of information from the 
system in question 

£ — Measure phase of DMAIC 
¢ Collecting this information can be its’ own 
problem 
¢ The TRIZ methodology provides techniques 
specifically designed to resolve measurement 
and detection problems 
— Su-field Mode (SFM) 


— 76 Standard Solutions 
— SFM Algorithm 


_ Measurement and Detection Standards from the TRIZ Methodology 


Ape 


INOIND) ING 
INSSOOINS 


Simplify, Perfect, Innovate 


Michael S. Slocum, Ph.D., T.Sc., M.B.B. 
Chief Innovation Officer 
Air Academy Associates 
Colorado Springs, CO 80920 
mslocum@airacad.com 


CLASS 4. MEASUREMENT AND DETECTION 
STANDARDS 


Class 4 Standards with Examples 


| CLASS 4: MEASUREMENT AND DETECTION STANDARDS 
co GROUP 4-1: CHANGE INSTEAD OF MEASUREMENT AND DETECTION 


STANDARD 4-1-1 


If a problem involves detection or measurement, it is 
proposed to change the problem in such a way, so that 
there should be no need to perform detection of 
measurement at all. 


Example: To prevent a permanent electric motor from 
overheating, its temperature is measured bya 
temperature sensor. If to make the poles of the motor 
of an alloy with a Curie point equal to the critical value 
of the temperature, the motor will stop itself. 


: CLASS 4: MEASUREMENT AND DETECTION STANDARDS 
> GROUP 4-1: CHANGE INSTEAD OF MEASUREMENT AND DETECTION 


STANDARD 4-1-2 


If a problem involves detection of measurement, and 
it is impossible to change the problem to eliminate 
the need for detection or measurement, it is 
proposed to change/detect properties of a copy of 
the object (e.g. picture). 


Example: It might be dangerous to measure the length 
of a snake. It is safe to measure its length ona 
photographic image of the snake, and then 
recalculate the obtained result. 


ee CLASS 4: MEASUREMENT AND DETECTION STANDARDS 
GROUP 4-1: CHANGE INSTEAD OF MEASUREMENT AND DETECTION 


STANDARD 4-1-3 


If a problem involves detection or measurement, and the problem 

cannot be changed to eliminate the need for measurement, and it is 

impossible to use copies or pictures, it is proposed to transform this 
| a problem into a problem of successive detection of changes. 


Notes: Any measurement is conducted with a certain degree of 
accuracy. Therefore, even if the problem deals with continuous 
measurement, one can always single out a simple act of 


measurement that involves two successive detections. This makes 
the problem much simpler. 


Example: To measure a temperature, it is possible to use a material 
that changes its color depending on the current value of the 


temperature. Alternatively, several materials can be used to indicate 
different temperatures. 


| CLASS 4: MEASUREMENT AND DETECTION STANDARDS 
= GROUP 4-2: Synthesis of Measurement System 


5 STANDARD 4-2-1 


If anon-SFM is not easy to detect or measure, the problem is solved 
by synthesizing a simple or dual SFM with a field at the output. 
Instead of direct measurement or detection of a parameter, another 
ge parameter identified with the field is measured or detected. The field 
to be introduced should have a parameter that we can easily detect or 


measure, and which can indicate the state of the parameter we need 
to detect or measure. 


F 
S ran | 
1 


[___— «=—SF?7 +S, 
ne 


Example: To detect a moment when liquid starts to boil, an electrical 
current is passed through the liquid. During boiling, air bubbles are 
formed - they dramatically reduce electrical resistance of the liquid. 


1 modified 


CLASS 4: MEASUREMENT AND DETECTION STANDARDS 
GROUP 4-2: Synthesis of Measurement System 


> STANDARD 4-2-2 
If a system (or its part) does not provide detection or 
measurement, the problem is solved by transition to an 


| internal or external complex measuring SFM, introducing 
easily detectable additives. 


Fy F 
yf a 1 
S) +—S, C8, Sy +S, 


Fy modified 


Example: To detect leakage in a refrigerator, a cooling agent is mixed with 
a luminescent powder. 


: CLASS 4: MEASUREMENT AND DETECTION STANDARDS 
> GROUP 4-2: Synthesis of Measurement System 


STANDARD 4-2-3 


If a system is difficult to detect or to measure at a given 
| moment of time, and it is not allowed or not possible to 
introduce additives into the object, then additives that create 
an easily detectable and measurable field should be 
Introduced in the external environment. Changing the state of 
the environment will indicate the state of the object. 


Example: To detect wearing of a rotating metal disc contacting 
with another disc, it is proposed to introduce luminescent 
powder into the oil lubricant, which already exists in the 
system. Metal particles collecting in the oil will reduce 
luminosity of the oil. 


CLASS 4: MEASUREMENT AND DETECTION STANDARDS 
GROUP 4-2: Synthesis of Measurement System 


STANDARD 4-2-4 


If it is impossible to introduce easily detectable additives in the 

ge external environment, these can be obtained in the 
environment itself, for instance, by decomposing the 
environment or by changing the aggregate state of the 
environment. 


Notes: In particular, gas or vapor bubbles produced by 
electrolysis, cavitation or by any other method may often be 
used as additives obtained by decomposing the external 
environment. 


Example: The speed of a water flow in a pipe might be 
measured by amount of air bubbles resulting from cavitation. 


CLASS 4: MEASUREMENT AND DETECTION STANDARDS 


GROUP 4-3: Improvement of Measurement Systems 


STANDARD 4-3-1 


a Efficiency of measuring SFM can be improved 
by the use of physical effects. 


Example: Temperature of liquid media can be 
measured by measuring a change ofa 
coefficient of retraction, which depends on 
the value of the temperature. 


: CLASS 4: MEASUREMENT AND DETECTION STANDARDS 
> GROUP 4-3: Improvement of Measurement Systems 


STANDARD 4-3-2 


ge If it is impossible to detect or measure directly the changes in 
the system, and no field can be passed through the system, 
the problem can be solved by exciting resonance oscillations 
(of the entire system or of its part), whose frequency change 
is an indication of the changes taking place. 


Example: To measure the mass of a substance in a container, 
the container is subjected to mechanically forced resonance 
oscillations. The frequency of the oscillations depends on the 
mass of the system. 


CLASS 4: MEASUREMENT AND DETECTION STANDARDS 


GROUP 4-3: Improvement of Measurement Systems 


STANDARD 4-3-3 


ge If resonance oscillations may not be excited in a system, 
its state can be determined by a change in the natural 
frequency of the object (external environment) 
connected with the system. 


Example: The mass of boiling liquid can be measured by 
measuring the natural frequency of gas resulting form 
evaporation. 


CLASS 4: MEASUREMENT AND DETECTION STANDARDS 


GROUP 4-4: Transition to Ferromagnetic Measurement Systems 


STANDARD 4-4-1 


Efficiency of a measuring SFM can be 
improved by using a ferromagnetic substance 
and a magnetic field. 


Notes: The standard indicates the use ofa 
non-fragmented ferromagnetic object. 


CLASS 4: MEASUREMENT AND DETECTION STANDARDS 


Se GROUP 4-4: Transition to Ferromagnetic Measurement Systems 


STANDARD 4-4-2 


e Efficiency of detection or measurement can be 
improved by transition to ferromagnetic SFMs, 
replacing one of the substances with 
ferromagnetic particles (or adding 
ferromagnetic particles) and by detecting or 
measuring the magnetic field. 


ee CLASS 4: MEASUREMENT AND DETECTION STANDARDS 


GROUP 4-4: Transition to Ferromagnetic Measurement Systems 


) 
STANDARD 4-4-3 
ge If it is required to improve the efficiency of detection 


or measurement by transition to a ferromagnetic SFM, 
and replacement of the substance with ferromagnetic 

particles is not allowed, the transition to the F-SFM is 

performed by synthesizing a complex ferromagnetic 


SFM, introducing (or attaching) ferromagnetic 
additives in the substance. 


wha CLASS 4: MEASUREMENT AND DETECTION STANDARDS 
GROUP 4-4: Transition to Ferromagnetic Measurement Systems 


STANDARD 4-4-4 


If it is required to improve efficiency of 
detection or measurement by transition to F- 
SFM, and introduction of ferromagnetic 
particles is not allowed, ferromagnetic 
particles are introduced in the external 
environment. 


CLASS 4: MEASUREMENT AND DETECTION STANDARDS 


GROUP 4-4: Transition to Ferromagnetic Measurement Systems 


STANDARD 4-4-5 


Efficiency of a F-SFM measuring system can 
be improved by using physical effects, for 
instance, Curie point, Hopkins and 
Barkhausen effects, magneto-elastic effect, 
etc. 


CLASS 4: MEASUREMENT AND DETECTION STANDARDS 
GROUP 4-5: Evolution of Measurement Systems 


» STANDARD 4-5-1 
Efficiency of a measuring system at any stage of its 
o evolution can be improved by forming bi- and poly-system. 


Notes: To form bi- and poly-systems, two or more 
components are combined. The components to be 
combined may be substances, fields, substance-field pairs 
and SFMs. 


Example: It is difficult to accurately measure the 
temperature of a small beetle. However, if there are many 
beetles put together, the temperature can be measured 
easily. 


CLASS 4: MEASUREMENT AND DETECTION STANDARDS 
GROUP 4-5: Evolution of Measurement Systems 


STANDARD 4-5-2 


Measuring systems evolve towards measuring the derivatives 
| a of the function under control. The transition is performed 
along the following line: 


Measurement of a function + measurement of the first derivative 
of the function + measurement of the second derivative of the 
function. 


Example: Change of stress in the rock is defined by the speed 
of changing the electrical resistance of the rock. 
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Agenda 


e Background 


e Systems Integration, Test and Performance Measurement 


e Summary 
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What is Computer Program Systems Integration, 
Test and Performance Measurement? 


e The ability to verify the interfaces, functions and measure 


performance of two or more computer programs on the 
target hardware suite and operating environment. 
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Catalyst for Early Computer Program Systems 
Integration, Test and Performance Measurement 


e Introduction of COTS hardware 
e Conversion of legacy software to new languages 


e System complexity with multiple hardware and operating 
environments 


Software Development Process 


Cor 
Ted es 


EXPERIENCE. RESULTS. 


System Definition 


Computer Program |Computer Program| Computer Program 


System Integration 


Reats Definition 


System 


PRIME 
Responsibility 
CSC 
Responsibility 


Reats Definition 


Element 
Reats Definition 
CP Performance 
Reats Definition f seeC>® 


HOR) Computer | 
Program Design ntegration & Tes 


& Design Definition & Design Implementation Testing Testing 
A A 
: Development Phases 
Operational System Requirements Analysis/Design 


Software Requirements Analysis 
Preliminary Design 

Detailed Design 

Coding & CSU Testing 

SDR CSC Integration & Testing 

CSCI Testing 

System Integration & Testing 


PDR 


|). 1 
Comp Program il } 
Detailed Design Unit Test 
FO 
CR j 


Element CP 


IPR f 


Operational 
Tests 


System Demo & 
System Qual Tesy 


System T&E, 
MEIT 
est & Evaluatio 


j 


yy. TPR 


Multi-Processor Environments 


e Legacy 
—AN/UYKs 

e COTS Processors 
— Single Board Computers (SBCs) 
— Symetric Multi-Processors (SMPs) 


COTS Processor OPerator User 
Cabinet Console Display 


Console 
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ANIUYK-43 
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Multi-Operating Environments 


e Legacy 

— Aegis Tactical Executive System (ATES) 
e COTS 

— Concurrent Powermax 

— Sun Solaris 

— Red Hawk Linux 

—LynxOS 


concurrent 39 Su. 
cS) redhat. @ | 


invent 
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Multi-Computer Programming Languages 


e Legacy 

—Compiler Monitor System -2 (CMS-2) 
e New 

—Ada 

— C/C++ 

—Java 
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Systems Integration, Test and Performance Measurement 
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Process Improvement 


e Incremental Systems Integration, Test and Performance 
Measurement during the software development process 


—Improve System Stability 

— Early Identification and Resolution of issues and defects 

— Significantly reduce Engineering Test & Evaluation failure rates 
— Processes and Procedures QA reviewed and approved 


CSC DMEI DE CMMI Level 5 
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Software Development Process 


System Definition Computer Program |Computer Program| Computer Program | System Integration 
& Design Definition & Design Implementation Testing Testing 
Development Phases 


Operational 
Regts Definition 


System Requirements Analysis/Design 
Software Requirements Analysis 
Preliminary Design 

Detailed Design 

Coding & CSU Testing 

SDR CSC Integration & Testing 

CSCI Testing 

System Integration & Testing 


Operational 
Tests 


System 


eae System Demo & 
Reats Definition MRP 


System Qual Test 


System T&E 
MEIT 


Engineering 
est & Evaluatio 


Element 
Regts Definition 6pB 


CP Performance 
Reats Definition 


PRIME 
Responsibility 
CSC 
Responsibility 
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Systems Integration 


e Validate and maintain operability of system hardware and 
operating environments 


e Verify Computer Program Interfaces 


laEcVae\ rc ec me @) =m @xe)anlolUics) au admelele-Vinm alec) ac-(er-m (ot-U( oto aot-ye) \V/cremeleralaleiiaal= 
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Systems Integration continued 


e Integration issues tracking and reporting 
— Integration Issue resolution prior to computer program delivery 


BUILD ISSUES = ACTIVE ISSUES 
ISSUES CLOSED BECAME BEING 
DURING BUILD ADDRESSED 


BUILDS 


Build 4 
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Systems Test 


e Development and Regression Functional Testing 
— Documentation 
e Plans 
e Procedures 
— Execution 
e Utilizing two or more computer programs 
— Multiple hardware and operating environments 


5% to 10% Improvement of Engineering Test & Evaluation 
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Systems Test continued 


e Test defect reporting and tracking 
— Test defect resolution prior to computer program delivery 


System Integration & Test 


OLow 
BMEDIUM 
DHIGH 


Capability 
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Systems Performance Measurement 


e CPU and Memory Utilization 
e Thread and Response Timing 
e Measurement Tools 
— Legacy 
e ATES Data Recording 
—COTS 
e UNIX TOP and Kernel Trace 
¢ Concurrent Nightview 
e LynxOS Spyker 
e Standard scenario 
—Function and information loading 
— Repeatable 
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Systems Performance Measurement continued 


e Performance Measurement Abnormalities 
— CPU and Memory Utilization Increase 
— CPU Utilization Spikes 
— Memory Leaks 
— Slow Thread or Response 
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Systems Performance Measurement — CPU Utilization 


UseriSysem CPU Utilization 


——3)s 


cen meee IS iy 


——— UsertSys 


Time (Aug-29-Run) 
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Systems Performance Measurement - Memory Utilization 


User Process Memory 
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Systems Performance Measurement - Memory Leak 


Free Memory 


Memory (Meg) 


Time (Aug-29-Run) 
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Systems Performance Measurement continued 


e Performance Measurement issues resolved before 
computer program delivery 


10% to 20% CPU Utilization Improvement 
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Summary 


e Incremental Systems Integration, Test and Performance 
Measurement 


e Integration and Test issue and defect resolution during the 
computer program development phase 


e Improved Computer Systems Stability and Performance 


Cost Effective by identifying and resolving systems issues and defects 
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- Panel - 
Systems Engineering and 
DT&E for Systems Suitability 


Colonel Rich Stuckey 
Principal Assistant for DT&E 
OUSD(AT&L) Systems and Software Engineering 


DT&E and suitability...do we 
have a problem? 


systemic DT&E Findings 
OSD Program Support Reviews 


e Maturing suitability in SDD is not a priority 


— Few efforts observed to design-in reliability 


Many reliability requirements lack a mission context 
— Maturation timeframes or maturity at IOC not defined in requirements 
— Log Demos to evaluate IETMs and diagnostics effectiveness rarely held 
e Log demos in PD phase are conducted too close to IOT&E 
¢ Most programs lack quantifiable MS C entrance criteria 


— Don't address R&M, manufacturing, integration, Net Ready, etc. 


e Many programs downgraded ACAT ID to ACAT IC at MS C 


— Not supported by demonstration of full capabilities, including suitability 


SE, DT&E and Suitability 


MS A MS B 


Joint System Development & Demonstration 
Baan ; ; Production & 
Planning Requirements Design, Readiness Deployment 


Analysis Review 


User Require- 


System 
ments & Concept x. 2D Demonstration 


of Operations & Validation 


Requirements Integration 
& Architecture & Test 


Component 


Concepts for Suitability era Integration 
Start early & Test 


Focus continuously 
Add operational realism 
Raise issues early 


Procure, Fabricate, 


& Assemble 
Parts 


DOD Guide for Achieving Reliability, 
Availability, and Maintainability 


DOD GUIDE FOR ACHIEVING 


RELIABILITY, AVAILABILITY, AND MAINTAINABILITY 


http://www.acqg.osd. 
mil/se/publications/pi 


IRAM%20Guide%20 
(080305).pdf 


Panel Format 


Initial remarks by each panelist 


Panel O&A 
Moderator asks Q 
1 panelist takes initial A - no time limit 
Panelist fire-at-will after initial A 
Moderator calls time at ~8 minutes 


Panelists 


Dr. David M. Jerome 


Deputy Director of Air, Space and Information Operations 


Headquarters, Air Force Materiel Command 


Mr. Richard L. Schubert 


Vice President and Chief Engineer 


Lockheed Martin's Integrated Systems and Solutions 


Mr. Brian M. Simmons 


Director, US Army Evaluation Center 


Mr. Ray Lytle 


Director of Life Cycle Engineering 


Raytheon Missile Systems 


Question 1 


How well do the systems being fielded 
in lrag meet their sustainability 
expectations? 


Question 2 


How do we resolve the conundrum: 


- The user drives rapid fielding (“tyranny 
of the urgent"); 

- But DOT&E raises the issue of 
sustainability, while rapid fielding 
bypasses a disciplined approach to 
Suitability. 


Question 3 


Do system requirements sufficiently 
address sustainment? 


Question 4 


How could we modify traditional DT & 
OT processes to improve 
sustainment? (and, how to evolve 
DT/OT to fit the rapid fielding 
process)? 


Question 5 


How is DT&E for software different 
from hardware for the suitability, 
effectiveness and sustainability 
arena? 


Question 6 


Can we link M&S used in suitability 
analyses, with M&S used in system 
performance analyses, so more 
complete and early decisions can be 
made for systems engineering? 


Question 7 


Do system development contracts 
Instruct industry sufficiently to 
design & deliver sustainability? 
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Cost Efficient Risk Management 
through 


Life-Cycle Integrated T&E 


Josh Tribble 


MILITARY ANALYST 
AVW TECHNOLOGIES 


Phone: 757-361-9587 

E-mail: tribble@avwtech.com 
860 Greenbrier Circle, Suite 305 
Chesapeake, VA 23320 
http:/Iwww.avwtech.com 


STRENGTH THROUGH INDUSTRY & TECHNOLOGY 


Agenda 


elntroduction 


— Acquisition humor 
— Complexity challenge = increasing risk 
— Intro to Integrated T&E 
eIntegrated T&E within systems engineering to manage risk 
— Alignment of T&E processes within systems engineering process 
— Integration of T&E organizations/processes throughout acquisition life cycle 


eEnablers to implement IT&E within a program 
— Risk based T&E planning and reporting 
— AVW IT&E Database Toolset 
— Other recommendations for implementing IT&E 


eConclusion/ Q&A 


NOTE: My remarks are intended to spur thought on improving how we as testers 
can do business better to support the warfighter. While | hope this aligns well with 
DoD and Services T&E initiatives, | am not representing any government agencies’ 
Official position. 


Acquisition 101? 


How the How the 
user described it | the requirement contractor programmer PM/sponsor 
was understood designed it wrote it described it 


How the project What was How the How the helpdesk | What the user 
was documented | actually installed | Government was supported it really needed 
billed 


How do we avoid this? 


A = VW Complexity 
TECHNOLOGIES 
Challenge 


eOpen Architecture/Systems 
«Complex C4I—GIG/FORCEnet 
«Joint Interoperability 
eEmerging Technology 
& Materials 
«Capabilities Based 
Requirements 
*CAIV 


| eMore difficult to develop 
: *More difficult to test 
«Compressed timelines 
«Compressed budgets 
eMORE RISK...& HIGHER COSTS 


met tte TMCS TIS pane 
ee oi a oa a 
0 a ac “AOE atte 50 OD ce 


DFvs-OF — IT 


DT OT 
e Test to specs. ¢ Operational environment & threat with 
e Limited test environment perhaps in end users & support 
lab e End-to-end mission perf. & Support 
¢e Focused on a specific set of criteria. | ° Production representative; system/ 
¢ Test threshold values not capability family of systems 
° Critical technical parameters ¢ Test overall capability of an item to 


¢ Integration testing designed around etree ieee al ete art cr 
minimum performance criteria and P 


interface spec. ¢ Test the limitations and capabilities of 


e May not address all threats or 20 MEW SO Wek: 
missions. ¢ Employ and assess doctrine/TTP 


¢ CT adds contractual issues * Independent IOT&E & LFT&E 
Controllable Factors mandates (Title X) 


Inputs (x) ue 
an —»| THIS MUST TRANSFORM INTO A 
-, a soo et le le el ta ek eI 
e Increasing fidelity of technical and 
operational assessments 
¢ Cooperating organizations 


¢ Reduced budget and timeline ? 
¢ Team/IPT structure not competitive 


Integrated T&E 


Key component of risk management for a program 


¢ Coordinated planning and development of 
CT & individual test objectives 


OL Bans ¢ Some data elements will not overlap and can only 
be measured in a technical, operational, or live fire 
OT&E Plan ; 
environment but there can be potential for 
significant cost savings and earlier risk reduction 


e Requires buy-in from all orgs + strong T&E WIPT 


LFi&E Plan 


OT&E 
DT&E 
NA CT | 
LFT&E 2 
Joint Exp, ACTDs 


Integrated T&E Plan 


System Disposal 
"ae (CT, DT, OT, LFT&E, Joint Exp, M&S, Analysis, etc.) dt 


Program Start 


Process 


'\ Be 


T&E 
The > Tat 


Loop 


Testers support by influencing: 

e Measurable, objective, meaningful reqs 

e Reqs context & op scenarios 

¢ Bounding system (technical/operational) 

e Assisting mission / functional breakdown 

¢ TPM selection 

¢ Influencing HSI 

¢ Prioritization of reqs (critical / need / want) 

¢ IV&V of reqs flowdown + delivered technical 
and operational capabilities 


T&E During 
Sys Eng Tasks 


THE SYSTEMS ENGINEERING PROCESS 


Design Loo XY 


Process 
Output 


T&E is supported by insight into various 
aspects of project to facilitate efficient 
test planning: 

e Customer expectations 

¢ Project & external constraints (CAIV...) 
Reqs context and intentions 

¢ Life cycle support planning 

¢ HSI planning/design 

e Physical / logical architecture drivers 

e Prioritization of requirements 


T&E During 
Sys Eng Tasks (Cont’) 


THE SYSTEMS ENGINEERING PROCESS 


Process 


Input SACU UEU BLS 
\ Requirements & Control 


Analysis (Balance) 
| Requirements aro) 
_ LOOP Allocation 


T&E Mie | \ 


TLE es > a BY LUE 
Loop —— 


Process 
Output 


Testers support by influencing: 
e Consistency in reqs/functional flowdown 
based on original intentions and op context 
e Influencing HSI in detailed design including 
user reviews of HCI & functionality 

¢ Verification of requirements implementation 
through limited component level tests 

e Interface definition 

¢ Prioritization of lower level requirements 

¢ IV&V of reqs flowdown + delivered technical 
and operational capabilities 

¢ M&S planning/development 


T&E supported by insight (which 


¢ Detailed reqs flowdown and prioritization 
¢ Detailed life cycle support planning 
¢ HSI planning/design 
¢ Detailed architecture drivers 

& early collection of evaluation data: 
¢ Life cycle planning 
¢ HSI design implementation 
¢ Software eng. process assessment 
M&S V&V 
¢ SCl/Component & interface test data 


improves test planning efficiency) into: 


T&E During 
Sys Eng Tasks (Cont’) 


Input System Analysis 
Requirements - & Control 


, 
(Balance) 


Analysis 4, 
' Funtional Analysis/ aaa 
L Allocation — 
Verification . 


T&E 
Feedback Tae fumes Synthesis 


Loop Process 


Output 


Testers support by influencing: 

¢e M&S analysis planning 

e Monitoring M&S development 

e Assisting in M&S analysis execution 

e Independent evaluation of analysis results 

¢ Evaluation of systems and software 
engineering process/process improvement 

¢ Independent review of risk management and 
input of T&E issues as new/updated risks 

e Objective TPM tracking 

¢ Design for safety, life-cycle, interoperability, 

& Survivability (instead of merely testing) 


T&E is supported by insight into: 

e Capabilities and limitations from analysis 
that points to need for live testing 

e Pre and post-test predictions 

¢ Test design and noise factors selection 
(design of experiments), sensitivity studies 

e System & component trade-offs 
& collection of evaluation data: 

e Analytical and M&S based evaluation of 
system performance 

M&S V&V 


T&E During 
Sys Eng Tasks (Cont’) 


THE SYSTEMS ENGINEERING PROCESS 


Process 


Input System Analysis 


\ Requirements & Control 
Analysis (Balance) 


Requirements Barn re CHT 
Loop Allocation 


Er Verification \, 
Feedback Synthesis 


| Loop 
a —< 


Process 
Output 


Testers support by: 

e Planning and executing tests to verify 
requirements and validate functions and 
mission capabilities 

¢ Giving engineers insight into performance of 
system 

¢ Independent internal and external agencies 
evaluation of the system 

[traditional T&E - with greater 
participation from systems engineers & 

increased use of standard engineering 

methodology for planning efficient tests 


T&E is supported by systems engineers: 

e Interpretation of technical results 

e Determining impacts on HSI, life-cycle 
planning, IA, etc. 

¢ Categorization of issues and problems 

T&E supports accurate decision making: 

¢ Proceeding with output to next acquisition 
phase, or 

¢ Proceeding to next phase of testing, or 

e Repeat of previous tasks while holding at 

this point in the acquisition cycle 


A\/ \W 


TECHNOLOGIES 


Systems Engineering + T&E 
within the Acquisition Cycle 


System Development 
& Demonstration 
Critical 
Design 
Review 


THE SYSTEMS ENGINEERING PROCESS 
System Analy 


Operations & 
Support 
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iequirements 
acation acation 

erification 
Synthesis Syothesis 
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MORE THAN TESTING ... 
CONTINUOUS EVALUATION THROUGHOUT THE LIFE CYCLE 


SION CAPABILITY DELIVERY... 
INTEGRATED T&E & SYSTEMS ENGINEERING 


Ability to influence System maturity & 


system design roC=X-JCe lay mel ele le-lelomexes-yt 


Systems Engineering + T&E 
within the Acquisition Cycle 


T&E eRequirements testability review eTechnology Feasibility 
ACTIVITIES ¢Concept studies and analysis *EDM DT 

DURING *CONOPS development Design Reviews 
C&TD eMission/Functional analysis *EOA 


PHASE «Joint Experimentation/ACTD/ATD eM&S 


Systems Engineering + T&E 
within the Acquisition Cycle 


T&E *Detailed Design Rvw_ eLab/Land Based Testing & M&S 
ACTIVITIES *eDesign OA eLimited field/at-sea testing & usability testing 
DURING *Detailed CONOPS eTestbed/surrogate LFT&E 

SDD eUser reviews eSupportability analysis/ initial logistics audit 


PHASE TTP development *RAM analysis & testing (reliability growth) 
eTrade studies *Certification process 


T&E 
ACTIVITIES 
DURING 
P&D 
PHASE 


Systems Engineering + T&E 


within the Acquisition Cycle 


*Production Readiness Reviews 
*Detailed CONOPS 

«Production H/W, S/W review 
Detailed TTP review/refinement 
eProduction in-process review 
w/ users & verification team 


*Federated Simulation Testbed 
Production acceptance + mission 
validation DT/TECHEVAL and lIOT&E 
eHSI, Manpower, Personnel, Trng Eval 
eSystem LFT&E & Shock Qual 

eFinal Independent Logistics Audit 


T&E 
ACTIVITIES 
DURING 
O&S 
PHASE 


Systems Engineering + T&E 


within the Acquisition Cycle 


eTTP refinement 
eSoftware/System problem 
characterization & tracking 
eUpgrade/ P3l / spiral development 
requirements analysis 


eTesting software upgrades 
*M&S/DT/FOT&E for system fixes / 
upgrades / new increments 

*RAM & supportability data collection 
eAdditional LFT&E/shock qual 


Enablers for implementing 
IT&E for risk-management 


¢ Actually implementing a process for IT&E with 
adequate buy-in is the first step 


¢Use software tools to step through planning 
and reporting processes and document IT&E 


eimplement risk based test planning and 
reporting 
eOther recommendations to follow... 


TsE Framework Toolset 


Test Results Tracking 
*Test Event Results 

Test Event «Program / Design Issues 
*Risk Categorization 


—_ Planner: 
E ACKING | fest Event Lessons 


Learned 


Test Design 
Test Matrix & Resource Planning 


-TEMP Inputs 


*Test Plan Inputs 


*Test Report Inputs (Results and Issues) 


*Tailored Reports for Issue Status to User 


Ts&E Framework Toolset Architecture 


TECHNOLOGIES 


System CDD Capabilities / Requirements Working Group 


Other References (System CONOPS, TTP, Specs) Affected COI / Test Obj 


Lo ail Test Event 


a Description [Mm Affected System (s) 


Priority Date Opened 


| UJTL/ Service Tasks / 


Risk Level Last Updated 


Issue POC Resolution 


7Nes ie) BJU (=m BY-li=) 


Action To Date 


Action POC SUS) 


Closed 


T&E Framework Toolset 
TECHNOLOGIES Mission Based Test Design 
How well is “it” 


What is “it” supposed to do it? 
supposed to do? 


| System CDD Capabilities / Requirements | 


OI rer errs es etl 
Other References (System CONOPS, TTP, Spex 


How do we test “it”? 


T&E Framework Toolset 


Functions/ Capabilities 


eReview of requirements, capabilities, 
mission tasks, or functional tasks 
«Development of COls/ Test Objectives 


«Develop MOE/MOS/MOP/CTP and 
trace to test objectives and source 
requirements 

eDevelop test method and data 

| requirements for each measure 


_| «Group test objective/tasks into testable 
~ blocks (vignettes) and determine 
resources 

eDevelop test matrix based on 
conditions 


T&E Framework Toolset 


Functions/ Capabilities (cont’) 


eldentification of test objectives 
and measures for a given event 
(exported from T&E Framework 
Tool) 


eResource and cost estimation to 
support TEMP, budget 
programming, test planning, and 
other efforts including ties for each 
a resource to test objectives. 


Risk Level 


Issue POC Resolution 


Action Due Date 
Action To Date 


Action POC SIGs} 


Closed 


T&E Framework Toolset 


Functions/ Capabilities (cont’) 


Traceability of test results to test event to 
objectives to parent requirements 

Risk based issue assessment 

Rapid reporting of issues 

Long term archiving of test results and 
program status 

User tailored reports to assess risks by 
function, mission area, system, req., etc. 


T&E Framework Toolset 


Functions/ Capabilities (cont’) 


| 


e Lessons learned tracking in 
standard Joint/Service 
Lessons Learned formats 

e Cross program visibility for 

a T&E process maturation and 

Se ES i. cost reduction identification 


nsure ram: d old m: 
anges ght pei eam/Pm rr 
ead ay a on gs were a seed idea and overall 

ig. Additi all ty. rs need to bet! anacesta ind focus of 


Risk Based 
TECHNOLOGIES Test Planning & Resourcing 
| 

[Probability of Occurrence —=~=~S~Comsequence | 5 | 4 | 3] 2] 1 


A — Frequently occurs during tests/operations (prob ~ 1.0) 


This supports 
B — Probably will occur during tests/operations TEMP test 
E — Not likely to occur during tests/operations (prob ~ 0) all ocall On 
detailed test 

Consequence Levels: . _ . planning; 
1: prevents accomplishment of primary mission or presents a serious safety hazard 
2: sig pri mission degradation w/o a work-around, secondary mission failure, or mod safety hazard removes 
3: major secondary mission degradation w/o work-around; pri mission degradation w/ work-around =| /7 uch of 
4: minor degradation/impact to primary and secondary missions subjectivity 
5: no impact to mission but operator annoyance or recommended enhancement surrounding 
Risk Levels: allocation of 
I: High Risk — The spec/req/capability req significant CT, some independent DT and OT; highest scarce 

pri for resource allocation; more test runs/ conditions permutations than other tests; most scrutiny : 

required before integrating tests testing 
II: Moderate Risk — Requires some dedicated DT and OT; medium resource priority; less scrutiny funding. 


before integrated tests completely 
III: Low/Manageable Risk — Little to no independence between CT, DT, OT, and LFT&E req; strong 
candidate for fully leveraging a small set of integrated tests for all data; lowest priority for resource 
allocation. 


AVW. Risk Based Test Reporting 


Probability of Occurrence Consequence See ek 
A — Frequently occurs during tests/operations (prob ~ 1.0) 
B — Probably will occur during tests/operations 


This could 

be tied 

directly 0 

risk register 
Consequence Levels: ane ee pots 
1: prevents accomplishment of primary mission or presents a serious safety hazard reporting of 
2: sig pri mission degradation w/o a work-around, secondary mission failure, or mod safety hazard CT, DT, OT, 
3: major secondary mission degradation w/o work-around; pri mission degradation w/ work-around |LFT&E, M&S 
4: minor degradation/impact to primary and secondary missions Runs, or any 
5: no impact to mission but operator annoyance or recommended enhancement 


other 
analysis or 
test 


Risk Levels: 

I: High Risk — resolve prior to fielding & conduct major re-test of mission area prior to fielding with 
the most resources applied 

II: Moderate Risk — resolve prior to fielding and re-test the specific requirement as soon as possible 
(depending on the requirement, re-test may be allowed to be conducted during follow-on T&E after 
fielding); apply moderate amount of resources to re-test 

III: Low/Manageable Risk — resolve when possible but does not impact fielding; re-test at next 

available previously planned test event; lowest prioritization for test resources 


Additional Recommendations 


¢ Fully implement IT&E top-down and institutionalize with PEO/PM orgs 


¢ closer align T&E Strategy/TEMP, Systems Engineering Management Plan, 
and Acquisition Strategy 


¢ Maximize test data and usage of that data across test programs and fully align 
results to the program’s risk registry 


¢ Conduct assessment and testing as early as possible and with all 
organizations to support risk mitigation 


¢ More test objective to requirements traceability in the TEMP 


¢ Service T&E reorganize to Enterprise business model to drive IT&E plus 
alignment with JT&E, DOT&E 


<a 


Additional Recommendations 


(cont’) 


¢ Implement more systems engineering rigor across T&E 
¢ Collect metrics on early risk mitigation efforts of TRE 


¢ Develop and field in consolidated baselines to reduce testing, integrate across 
programs not just within 


e Stress to threats and operating environments early and often 

¢ Change T&E score-card to a risk assessment vs. capabilities; continuous 
feedback throughout tests; foster more cooperation including leveraging 
JT&E, Experimentation, Training Exercises 

¢ Increase PM focus on life cycle, HSI, other factors beyond technical mission 
performance 

¢ Coordinate use of standard statistical methodology for T&E including Design 
of Experiments 


(Note—similar presentations made to 05 & ’06 NDIA Sys Eng Conferences) 
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T&E Rapid Response to the Warfighter 


¢ How does the T&E community, as part of 
the systems engineering process: 


— Obtain, maintain, and increase its body of 
knowledge, to assess and project with greater 
confidence, a product's potential capability 
and suitability (A,, Reliability, Deployability, 
Sustainability) during product development as 
the product and requirements change? 


¢ Knowledge not limited to quantitative data 


“Traditional” T&E Process 


Sa 


MS B OTRR 
Subsystem System System Limited Mission Measures 
Attributes Attributes Performance Mission Integration Assessment 
(KPPs) Performanc 
| Configuration 
CH) 
2p | Changes and 
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: | 
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Na 


DT/OT OT/DT | OT 
| (Capstone Test) 


Planning CT Test D 


Testing Over Time 


Sse raion 


Integrated Test 
Knowledge Over Time 


MS B OTRR 
Subsystem System System Limited Mission Measures 
Attributes Attributes Performance Mission Integration Assessment 
(KPPs ) Performance 
Potential levels 
| of granularity: 


| Point 
- Observations 


Knowledge 


Planning CT Test DT DT/OT OT/DT | OT 


fx Several Benefits to Dynamic System T&E 
Greater depth and breadth of knowledge 


Greater understanding of impact to 
changes 


~ « Greater understanding of test cost 


¢ Greater ability to assess product capability 
at points in time. 


ge Several Challenges to Dynamic System T&E 


e Transparency — ability to see across 
boundaries 


Permeable boundaries — ability to pass 
information and data (transportability) 
across boundaries 


e Product configuration management 


Graduated Reliability during development 
— Thresholds (maturity measures) 
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e Overview 
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ARMY TEST AND EVALUATION COMMAND 


Background 


DOD 5000.1 — “The primary objective of Defense acquisition is to 
acquire quality products that satisfy user needs with 
measurable improvements to mission capability...” 


Joint Capabilities Integration and Development System 
e War Fighting Capability Gaps 
e Material/Non-material Solutions 


Materiel System Operational View — mission tasks, activities, operational elements 
Performance Attributes and information required to accomplish warfighting mission. 


Key Performance Parameters 
System View — system elements and capabilities necessary to 
support warfighting functions. 


Technical View — set of rules and standards to ensure that a 
system satisfies a set of operational requirements. 


av | All View — overarching architecture that supports the OV, SV and 
TV. 
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ARMY TEST AND EVALUATION COMMAND 


Introduction 


T&E Process Paradigms: | | Traditional; | | Proposed 


Evaluate Mission 
Performance 


Completes the Feedback Loop to Mission Needs 


ARMY TEST AND EVALUATION COMMAND 
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Overview 


The Four Elements 


Mission 
Perspective 


SYSTEM ELEMENT 
System and Sub-system Functions 


TRE 
Perspective 
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EVALUATION ELEMENT 
Mission Ability and System Capability Measures 
TEST ELEMENT 

Data Products and Data Sources 


ARMY TEST AND EVALUATION COMMAND 


Overview 


Elements, Interfaces and Traces 
ELEMENTS AND TRACES 


Elements 
e Mission, System, Evaluation, 
and Test wenn EVALUATION TRACE 


PLANNING TRACE 


ELEMENT TO ELEMENT 
Interfaces INTERFACE 


e Mission to System 
: : SYSTEM 

e Mission to Evaluation ELEMENT 
e System to Evaluation 
e Evaluation to Test 
Traces 
Planning = Mission to Test 
Evaluation = Test to Mission ’ccoscnecece wii é 
Two Types: 
e Type 1 links Mission, System, Evaluation and Test Elements. 

¢ Plans and evaluates mission task ability through system function capability. 
e Type 2 links Mission, Evaluation and Test Elements. 

« Plans and evaluates mission task ability directly. 


EVALUATION 
ELEMENT 
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ARMY TEST AND EVALUATION COMMAND 


as Element/Interface Development 
Mission Element 


Purpose 
e Lo describe unit mission and tasks. COI: How capable is the (unit and system) in supporting Level 0 (Mission) 
; : i i (mission statement) in an operational environment? 
-- A task is defined as a discrete action that 
the unit (system and its operators) must 
perform in order to accomplish its mission. 
Level 1 (Task) 


Components 

e Critical Operational Objective: Mission : 
based — “How capable is the (unit and Level 2 (Task) 
system) in supporting (mission statement) |: 
in an operational environment.” 


e Task Levels: Orderly breakdown of the 
mission into tasks and sub-tasks. 


e Alternate Mission Tasks: Optional 
mission tasks used to accomplish part(s) 
of the mission. Alternate task options define different “mission threads.” 
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ARMY TEST AND EVALUATION COMMAND 


Element/Interface Development 
Mission Element — Example 


Development Keys 


e Temporal Format. COI: How capable is the UAS equipped unit in supporting the Commander’s RSTA and Level 0 (Mission) 


Armed RSTA requirements in an operational environment? 
« Temporal format provides a block diagram of 

mission to mission tasks in order of their 

occurrence. 


¢ Supports development of mission threads. 


Level 1 (Task) 


e Lowest Level of Mission Tasks. Level 2 (Task) 
« Lowest level mission tasks must be measurable. 


¢ Evaluated directly or indirectly via 
evaluation of system function capability. 


Level 3 (Task) 


e Support Documents. 


« Mission Need Statement, Initial Capabilities 
Document, Operational and Organizational 
Plan, Universal Task Lists, Capabilities 
Development/Production Documents 
(CDD/CPD). 

¢ Integrated architecture products in CDD/CPD 
uniquely support mission element. 

= OV-1: Who, How, Where, When, Why of the system and its mission. 
= OV-5: Operational activities (mission tasks). 


= OV-6c: Association of capabilities with sequences of operational activities (mission tasks). 8 


ARMY TEST AND EVALUATION COMMAND 


Purpose 
e To describe the system and the system 
functions. 


Components 
e System Items: Makeup of the system 
and sub-systems. 


e System Functions: Description of the 
function an item must perform in support 
of the mission. 


e System Level: Level of systems, sub- 
system, and components from the system- 
of-systems perspective. 


as Element/Interface Development 
System Element 


System of Systems 


1.1.F1 
Function of Sub- 
system 1.1 


1.2.F1 


Function of Sub- 


system 1.2 


Cnfelell 


Function of Sub- 


system 2.2 


Level 1 
(System/Sub-system) 


cco i a 
1S' Function of 


Sub-system 3.1 


3.1.F2 
2°4 Function of 
Sub-system 3.1 


2.1.1.F1 


Function of Sub- 


system 2.2.1 


Level 2 
(System/Sub-system) 
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ARMY TEST AND EVALUATION COMMAND 


ON Element/Interface Development 
re System Element — Example 


Development Keys 


e Item to Function Link. 
¢ Objective is to define the system functions. 


System of Systems 
Remote 


« System item is the sub-system responsible Se 
for providing the function. er ents 
LiF 2EM 
a = Cc icat C icat (S.LFL ; 
e System-of-Systems. pas poner cudeand 
¢ Include systems that are not part of the 2.1.F2 
2 . Communicate 
system being developed and evaluated if L2Fl with RT 
they are required to support the mission. sails a 
« Lowest Level of System Function. 
¢ Should be associated with the accomplishment rae 
of a mission task. 
SR Diol 
« Measurable by T&E. Send Fly to 
é Messages Waypoint 
e Risk Areas 
? = Level 2 
¢ Items and functions can be based on a specific (System/Sub-system) 
area of developmental risk. ane 
Zo ee 
e Support Documents. Control To ares 


and Landing 


¢ System Work Breakdown Structure 


¢ Integrated architecture products in 
CDD/CPD uniquely support mission element. 


= SV-1: Systems required to support the mission and the interfaces between them. 
= SV-4: System functions required to support the operational activities (mission tasks). 
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ARMY TEST AND EVALUATION COMMAND 


Element/Interface Development 
Mission to System Interface 


Purpose 
e To describe how the mission tasks MMMM MOIS 
° System 1 (AND/OR) - (AND/OR) (AND/OR) 
relate to the system functions. 7 LM | 
F ie a ey Sub- + (ANDIOR) | 4 (ANDIOR) 
system 1.1 = litior I Conditions i 
Components ee np Rule 


Function of Sub- 


e Mission Tasks: Taken from the Saul | 
— mo | as | ama 


(AND/OR) 


2; =| aS 
e System and System Functions: system 2.2 | Conditions _| 


2.1.1.F1 OEE) Rule 
Taken from the system element. Function of Sub ka UES I 
a Input Rule | 
e Input Rule: Description of how | as 
the system and its functions relate to cae 


Sub-system 3.1 Conditions 


the mission task. Uses logical input 
rules, such as AND and OR to describe links to more than one system or Ancien 


e Conditions: Description of the physical, military, and civil variations that effect 
performance of a task. For example; weather conditions, countermeasures, urban 


environment, etc. 
11 


ARMY TEST AND EVALUATION COMMAND 


Element/Interface Development 
Mission to System — Example 


Development Keys 
e Input Rule. 


¢ Link every function required to support _ 
the mission task. BRO commons 
¢ Link alternate system functions that ~[ aeuncions | AllEuncions [arr | [al Functions 
support the mission task. COMME ZB FLD 
« Top row = every oh ere if the system i ee | wihay | zAvamae | 02 | oe oz | oz oe 
: et OR 1.1,F1_ | OR 1.1.61 | | | OR LLF1 
suparts the mission tsk wih anti | et ene [oem 
¢ Linkages are important since they will be . a See 
used to evaluate mission tasks based “Seaton | Navigate AV 3 awienning 


on the evaluation of system functions/suitability. : | 4 Terrain 


| All Functions —_| 


2H) 23IFL 


e Conditions. 


air | Fly to 2. Flight Profile 
. eote ae Vehicle Waypoint +3. Day/Night 
¢ Consider the conditions based on the ability to —_ | 4 Wester dar 
support the mission task, but... Control ere al 
+c . . ‘ Takeoff and 2. | Length 
« The specific function may drive the choice Landing 
of applicable conditions. For example; terrain Eun 
may effect the communication functions of | | | EEE eee 
4 . . Wesile Guide and Hit ns z are 1908 
line-of-sight systems but not effect satellite Target eee 
systems. 


e Support Documents. 
¢ Initial Capabilities Document and System Threat Assessment Report to determine conditions. 
« Factors of METT-TC to determine conditions. 
¢ Integrated architecture products in CDD/CPD uniquely support mission element. 
= SV-5: Maps operational activities (mission tasks) from the OV-5 to the system functions from the SV-4. 12 
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Element/Interface Development 
Evaluation Element 


Pur OSe COI: Does the (system) perform (system capability)? 
e To describe the evaluation measures | Z 
and how they relate to mission tasks, oo BG acese 
system functions, and system suitability. —- ——_§_}— 
tptetes || | was | Ae aoe 
C t LLFLED arin | Ayer | | eunction ot | 
omponen S - 1.1.F1.E2.P1 Standard for ae ee per es 
e Conditions: Conditions are assigned to tasks Mor for MOr Lipie2 | Function 
that are linked directly to a MOE in the i2F1e1 || i2ra Input Rule 
: i ai Seuseueliel | “Wwree Gar Function of (AND/OR) 
evaluation element. MOP for MOE eee | Sanction bossaS 
e Aled bal Sal Aly =aL [aL 12F1 ae Conditions 
e Measure of Effectiveness (MOE): Parameter aor VU 
used to evaluate the system function or i (ANDIOR) 
mission task. 2.2-FLE1PL AIG 
ahi]; Oo FreEL Standard for] “Moeter | | Function of OI) 
e Measure of Suitability (MOS): Parameter used |—s3éterea ee | suc-cystem Se 
to evaluate the suitability of a system. mS eer ee Ba ees 
e Standard: Acceptable performance of the Capea Standard for 0.2161 ae 
Pp Pp Conait 
: Saas : ICIP ier WOVE MOE 0.2.1.E1| MOE for Task 01 ee raeres 
system function or mission task in terms of the 0.21.61 
MOE or MOS. 0.n.E1.P1 Standard for ae 
MIST thar Mlol= ee MOE for Task 1.2.F1 cobamens 
e System-focused COI: COI focused on O.n.E2 oes 


system or sub-system performance. Typically 

stated, “Does the (system) perform (a specific required capability)>?” 

e Link to System-focused COI: Column in the evaluation element that identifies which MOE/Ss are 
used to evaluate the system-focused COI. 


e Measure of Performance (MOP): Quantitative or qualitative measure of system performance under 
specified conditions. 
13 
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ON Element/Interface Development 
== [Evaluation Element — Example 


Development Keys 


e Mission and System Elements. 


KEY 
« All system functions must have at ele conor gd oad 
intended operating environment? EVALUATION 
least one MOE. 
¢ Mission tasks linked directly to a Measures of Ferornanes suncds | _ MOEMOS 
MOE usually indicate a need for i ee we ; 
evaluation during OT&E. messages | | SEMR<S | aaere2orapou | Ground] gyn 
1.1. F1L.E2.P3 Time of drop out. (Attribute) eid core 
« MOEs, MOSs and MOPs. ee en 12 ay 
fee <a via waypoints. Changes 
¢ System functions and mission tasks [susuereraiwes | | >100msqcer) | sasimremer —_| 
may have more than one MOE. 22FxELPLSionay Taoets | | fone) | Targets Detected Detect Target 
23.FLELPL Difference < 10 sec from 2.3.F1.E1 Waypoint 2.3.F1 
peter, Callares| euslenauel esumated ne rival On-Time % Fly to Waypoint 
¢ MOEs may have more than one MOP. ||| Limestarva. ot arval (aa) | 
2.3.1. FL.EL.P1 % of Successful Must Control 2.3.1FLE1% of 
TIO YES/NO (AA) Successful T/O 2.3.1.F1 Control 
¢ Both systems and sub-systems may 2a1FLEZPiwaSumeem | | Moxcowa | 294FLEDWG an 
h M O S S Landings ; SESS (AA) Successful ovis : 
ave one or more : 5 : 
3.0 
e “Dry run” evaluation from MOP 3..F1.ELP1 % targets hit. sro AGM Pray ot Sing ao Guise and Hit 
to mission task to ensure evaluation ileal A ee) 0.1.E1 % of Successful Mission Planning Sessions 
. 0.1,E1.P2 % successful loads. Judgment 
1S sound. Resse usoeiala Km(KPP) 0.2.3.3.E1 % of Targets Detected 


e Standards. 


¢ Assign a standard to each MOE to assist in resolution of the MOE. Typically four types of standards: 
= Direct Measurement: Compare demonstrated performance to standard. For example; maximum range. 
= Pass/Fail: Demonstration of a particular feature. For example; required number of hard points. 
= Comparison: Compare performance of two systems. For example; “performance equal to or greater than...” 
= Military Judgment: No specific standard. Military utility will be determined after the evaluation. 14 
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Element/Interface Development 
Test Element 


Purpose 

e To describe the data products, the 
sources of the data products, and 
how they relate to the evaluation 
element’s MOPs. 


Modeling 
and 
Simulation 


Developmental 
Test 


Contractor 
Test 


Data Product #1 


Data Product #3 
Data Product #1 
Data Product #2 


Data Product 
Data Product #2 


Components 
e Link to MOPs: Description of 


which data products support which 
MOPs. 


MOP 
TERE P A 
MOP 
2 Rep 


MOP 
CPAP sen [2 


MOP 
0.2.1.E1.P1 
MOP 
0.n.E1.P1 


Sx 
| 


Ee 


e Data Products: Specific data packet 
obtained though a data source 
satisfying a MOP data requirement. 


e Data Sources: The specific source of a data product. 
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Element/Interface Development 
Test Element — Example 


Development Keys 
e Data Products. 


2 # 5 5 5 
E 3 ® 5 i gee EVALUATION 
¢ Data requirements for each MOP are 5B 5 5 | 2 |88s 
translated into the data products. > allel | 8s AEE 
‘ — Ale el S|S|e} oF |S] 8125] oF) of 
« Requirements should be of sufficient fl aleebde alsa] als ede lh 
detail to provide the scope of the effort ale i a ila i a aaa a 
that will generate the data product. TLFLELPI of accuaIe=eH | | ~ oop pp) | LEFLED Da 
messages. Accuracy 
« Each MOP must have at least one data a ee s0%<% | sartezoep ou 
pr oO du ct. 1.1.F1.E2.P3 Time of drop out. aac 
1.3.F1.E1.P1 % via direct route. ili 1.3.F1.E1% of 
« More than one MOP can be supported | sarisivivvewyona | | owen | Stzustvcore 
by a data product. [sisirierawes | [>iwonsaon) [ sesinroner | 
2.2.F1,E1,P1 Stationary Targets reer = Soha f 
betwecn estimated and acted | | eciintedtime | 22°1E1Waypoin 
e Data Sources. vas time of arrival. of arrival (AA) 
2.3.1.F1.E1.P1 % of Successful Must Control 2.3.1.F1.E1 % of 
° TIO YES/NO (AA) Successful T/O. 
A Data sources can include: contractor 2.3.1.F1.E2.P1 % of Successful Must Control 23.1. FLE2 % of 
tests, developmental test, operational — a 
tests, field exercises, and modeling ae 


LEVEL 
3.1,F1.E1.P1 % targets hit. similar to AGM- Probability of Single 
XK hot Hit 


and simulations. 


0.1.E1.P1 Time to plan. Mi 0.1.E1 % of 


Successful Mission 
Planning Sessions 


lilitary 
0.1.E1.P2 % successful loads. Judgment 


- Evaluation Strategy. caaagimmamens | | agen | ozaaeina 
« The test element describes an 


integrated test program. 
¢ The test element also provides a method to view the acceptability of the entire evaluation strategy. 
= Are the data products sufficient to evaluate the MOE/MOS standard? 
= Which functions/tasks are demonstrated solely in DT? 
= Are there any functions/tasks that are not demonstrated prior to OT? Is this acceptable? 
16 
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Element/Interface Development 
Mission Test & Evaluation Plan 


e Documents the four elements and the interfaces between them. 


e Two main body chapters: mission evaluation and data sources. 


MISSION EVALUATION CHAPTER DATA SOURCES CHAPTER 


Mission Data Sources 
Description of the overall mission. Summary description of all data sources. 
- Mission Task Summary data product schedule for all 
Description of the mission task. data sources. 
System functions input rule. - Data Source 
Conditions. Purpose and description of the data 
- Measure of Effectiveness source. 
Description of the MOE. Scope and schedule of the data 
Evaluation Design and Procedure. source. 
Standard. - Data Products 
- Measure of Performance Description of the data product. 
Description of the MOP. Listing of the MOPs requiring the 
Method of Analysis. data product. 
- Data Product (s) 
Listing of required data product (s). 
- System 
- MOS; MOP; Data Product (s). 


- System Function 
- MOE; MOP; Data Product (s). 
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Application 
Test and Evaluation Elements 


e Test Element: 
¢ Data is collected from the data sources. 
¢ Data is then authenticated in terms of quantity, quality 
and applicability. 
¢ Authentication body (Data Authentication Group) 
includes representatives from the test events, other data 
sources, the evaluator and materiel developer. 


e Evaluation Element: 
¢ Data is then organized and analyzed. 


¢ Each MOE/S is rated as met or not met based on the 
standard. ” 
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Application 


System Element 


e System function capabilities and limitations are determined at the System Element. 
¢ Capability: “The (system) has the capability to (function capability with reference to standard).” 


¢ Limitation: “The (system) is limited to (function capability) which is (shortcoming with reference 
to the standard).” 


System Function Aggregation Tool 


END 
Document System 
Function Capabilities & 


e MOE/MOS ratings are 
applied to the system 


functions to determine the - END eens 
system capabilities and BONO System function is 
limitations. nn 

e Capabilities and limitations 
of lower level system level aystem All measure level system mm Determine 
functions are also used to resolved? Cae tations? a 


evaluate higher system 
functions. 


END Determine Determine 
System function is system function system function 
limitation. limitations. 
Unresolved 


e Tool developed to resolve 
the system functions. 
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Application 
Mission Element 


e Mission task abilities and restrictions are determined at the Mission Element. 
¢ Ability: “The (unit) has the ability to (task ability) while (task).” 


¢ Restriction: “The (unit) is restricted to (task ability) while (task) which is (shortcoming to 
mission task requirement if available).” 


Mission Task Aggregation Tool 


e MOE/MOS ratings are 
applied to the mission tasks 
to determine the mission 
abilities and restrictions. 


END 
Document Mission 
Task Abilities and 

Restrictions 


ae All END 
e System function capabilities Meee es ee Mission task is : 
and limitations are used to resclvedz Unresolved Peer eee 


abilities. 


determine mission abilities 
and restrictions. 


e Abilities and restrictions 
of lower level mission tasks 
are also used to evaluate 
higher mission tasks. 


All lower 
level mission 
tasks 
resolved? 


Any system 
function 
limitations? 


Any lower 
level mission task 
restrictions? 


All measure 
standards met? 


END Determine Determine Determine 
e Tool developed to resolve Aer ae mission ask mission task miseion task 
the mission tasks. Unresolved restriction. restriction. . 
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Application 
Mission Evaluation Report 


e MER provides the documented results of the evaluation. 


« Mission Evaluation Results. 


= Mission Performance in terms of Mission Threads. 
= Overall Mission Abilities and Restrictions. 
= Individual Mission Task Abilities an Restrictions. 
¢ System Evaluation Results. 
= System Performance in terms of Attributes and KPPs. 
= System Suitability 
= Overall System Capabilities and Limitations. 


Provides the decision maker with... 
a clear picture of the system capabilities and limitations allowing acquisition decisions 
based on the military utility gained. 


Provides the warfighter with... 
a Clear picture of the unit’s abilities and restrictions within the context of the mission. 


ARMY TEST AND EVALUATION COMMAND 


Weaknesses 


e Process is time consuming to plan and execute. 
¢ Requires extensive planning effort across functional boundaries (user, 
materiel developer, T&E). 


= “Sharing the burden” of developing the different elements with user, materiel 
developer and tester/evaluator can mitigate the impact. This also develops a 
consensus of the T&E strategy. 


= Database application software can be used as a tool to facilitate organizing 
elements and interfaces. 


« May require interpretation of results to determine capabilities/limitations and 
abilities/restrictions. 


= “Sharing the burden” again can be used. This develops a consensus of the results. 


e Not all information required to develop the elements is available at 
early system development milestones. 
¢ Systems in development prior to Milestones B may still be in competition. 


= Defining system items and functions in a generic sense can be used. System 
design specifics would be added after contractor selection. Also, generic system 
functions supports evaluation of technological risks. 
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Strengths 


e Provides a mission-based form of evaluation. 
¢ Military utility of the system immediately apparent to the user. 
¢ System suitability directly linked to mission capability 


e Outlines a fully integrated test and evaluation program. 


¢ Promotes synergistic use of data gathered from all sources: contractor test, 
developmental test, operational test, and modeling and simulation. 


¢ Promotes early identification of T&E strategy risks. 


e Provides continuous evaluation of the mission throughout all 
system development phases. 
¢ Impact of development risks on the mission visible in early development. 


« Monitors progress of system development and demonstration within the 
context of mission abilities provided. 


¢ Incremental development strategies are supported by evaluating each 
increment’s abilities in the context of the overall mission. 
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Summary 


e Mission-based evaluation process has been developed to support 
T&E planning and execution. Process is comprised of: 
¢ Four elements. 
= Mission Element: Comprised of the mission tasks and sub-tasks. 
= System Element: Comprised of system items and functions. 
= Evaluation Element: Comprised of the evaluation MOEs and MOPs. 
= Test Element: Comprised of the data sources and products. 
e Interfaces. 
= Links between each element have been developed to facilitate T&E planning and 
execution. 
e Execution of the T&E effort provides: 
¢ the decision maker with a clear picture of the system capabilities and 
limitations allowing acquisition decisions based on the military utility gained. 


¢ the warfighter with a clear picture of the unit’s abilities and restrictions 
within the context of the mission. 24 
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Element, Links & Traces 


ELEMENTS AND TRACES 


PLANNING TRACE 


sennenee EVALUATION TRACE 
ELEMENT TO ELEMENT 
INTERFACE 


SYSTEM 
ELEMENT 


EVALUATION 
ELEMENT 
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Acronym Chart 


AA Additional Attribute MOS Measure of Suitability 

AV All View (slide 4) OA Operational Area 

AV Air Vehicle (slides 11, 13, and15) OT Operational Test 

CDD Capabilities Development Document OT&E Operational Test and Evaluation 
COI Critical Operational Issue OV Operational View 

CPD Capabilities Production Document RSTA Reconnaissance, Surveillance & Target Acquisition 
DAG Data Authentication Group RT Remote Terminal 

DoD Department of Defense SATCOM Satellite Communications 

DT Developmental Test SV Systems View 

GCS Ground Control Station T&E Test and Evaluation 

JCIDS Joint Capabilities Integration and Development System T/O Takeoff 

KPP Key Performance Parameter ™ Telemetry 

MER Mission Evaluation Report TV Technical View 

METT-TC Mission, Enemy, Terrain, Troops, Time and Civil UAS Unmanned Aerial System 

MOE Measure of Effectiveness UAV Unmanned Aerial Vehicle 

MOP Measure of Performance 
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Thank you for that nice introduction. It is indeed a pleasure and an honor to 
be here today and to contribute to this ongoing conversation about how all 
of us can work together to improve the products and services we provide our 
nation’s defenders and first responders. Underwriters Laboratories has a 
long history supporting national defense and homeland security. We also 
have a long history with many of the companies and organizations here 
today, both in the private sector and public security arenas. At the same 
time, we have little direct NDI experience and our Homeland Security 
business is relatively new so what | can best do today is to share with you 
our business model for our Core Business and hope that there is a Best 


Practice or two than can be directly applicable to your work. 


Our security division was founded in the 1920s during the gangster wars in 
Chicago, when we published standards for stronger safes, more secure locks 
and bullet-resistant glass. During War World II, UL was classified as an 
“essential industry”. We tested devices to protect US plants from sabotage 


and trained military personnel in fire protection and general safety. 


Today, we continue to work with NDI affiliates, auditing security systems at 
facilities, undertaking basic research to advance the safety and training of 
first responders, and, of course, testing and certifying products that 
contribute to the safety and well-being of those who protect us and the 


public at large. It is a part of our business of which we are very proud. 


As most of you know, UL is an independent, not-for-profit product testing 


and certification organization dedicated to public safety. For over a hundred 


years, UL has played a central role in creating and maintaining a culture of 
product and public safety in the marketplace — a global culture that extends 
to designers and engineers, manufacturers and their suppliers, wholesalers 


and retailers, and to the consumers who purchase these products. 


Today, | would like to explain to you how this safety model works and share 
with you some valuable lessons we've learned over the years. | will also 
suggest ways this model could be applied throughout defense and homeland 
security, to assure quality and sustainability in a competitive market. In 


fact, | do think that “quality and sustainability” go hand-in-hand with safety.. 


UL first got into the testing and certification business over a hundred years ago. It 
all started at the 1893 World’s Fair, the Columbian Exposition in Chicago. One of 
the main attractions at the exhibition was the Palace of Electricity. This was a big 
deal. For the first time in history, the harnessing of this new technology called 


electricity was being displayed for the entire world to see. 


The problem was, fires were breaking out all over the place. Frequently. So the 
company that was insuring the Palace of Electricity called in a young engineer from 


Boston, William Henry Merrill to address the problem. 


Merrill brought with him a set of “best practices” for wires, switches and insulation 
that he applied throughout the exhibition, which, in turn, became a huge success 
in showcasing this new technology. Pretty soon manufacturers found out that the 
young engineer was doing this kind of work, so they would bring their products to 
him to make sure they were safe. That’s how UL got started and how William 


Merrill became our founder and first president. 


| mention our founding because it clearly demonstrates the natural formation of an 
environment that ensures quality, sustainability and safety. On the one hand you 
had manufacturers who wanted to sell this new technology, but on the other hand 


you had the authorities having jurisdiction — the insurance companies, the 


building owners, and ultimately, the end-users, who needed to know that the 


product works safely. As an independent voice, UL was able to bridge that gap 


and facilitate the market. 


The key, then, to creating and maintaining a culture of product and public 
safety is the complete participation and collaboration of all members of the 


marketplace. 


Essential, too, is the right private-government partnership. In the US the 
Consumer Product Safety Commission has broad latitude to mandate safety 
standards and testing. However, they have always had the perspective that 
voluntary industry compliance to standards and independent third party 
testing is preferable to government mandated standards and testing; always 
with the ‘big stick’ of government mandate should industry fail to do the 
right thing. Thus, CPSC always states their policy that compliance to 
standards and independent conformity assessment testing is ‘Voluntary but 
not Optional’. It is an excellent working model that protects the public, 
ensures a level playing field in the market, and leaves business unfettered to 


innovate. 


The critical elements involved in assuring product safety in a competitive market 


are: 


the development of safety standards to protect those who use the product. 


conformity assessment systems which test products against those 


standards and certify that they are compliant. 


Rigorous factory follow-up services to confirm continued compliance after 


certification is obtained 


e rigorous anti-counterfeiting efforts to protect the certification, the 
manufacturer who invests in it, the authorities having jurisdiction who 


demand it, and the end user who depends on it. 
e and finally, education and training throughout the marketplace. 


You can think of standards as the ‘rules of the game’ and conformity assessment 
and follow-up services as the ‘referees’. If you imagine a football game with no 
rules and no referees then you have a good picture of what many commercial 
markets would look like without standards, conformity assessment and follow up 
services - especially in a rapidly globalizing market with many emerging country 


participants/ 


Let’s take a closer look at each of these elements and the role they play in the 


safety model. First, the development of standards. 


A standard is a set of requirements or codes to improve a product’s safety and, in 
some cases, efficacy as well. A standard takes into account product usage, the 
environment under which it will be used, and the harsh conditions it may 
encounter and still remain functional. This is true for the motor of a vacuum 
cleaner, the sprinkler head of a fire Suppression system or the electrical safety of a 
string of decorative lights - one that can work in a Minneapolis Christmas and a 
Miami Christmas. Standards can be broadly encompassing - for example a DVD 
Player standard would include electrical safety, mechanical safety to ensure that 
small fingers don’t get injured by the moving parts, and EMC emission and 


susceptibility - perhaps performance testing too. 


Standards allow manufacturers to confidently invest in product development 
knowing they will have a market that trusts the safety and efficacy of their 
products. Standards increase competition and encourage innovation. Standards 


level the playing field. When all products meet certain requirements, price and 


special features become the points of differentiation, rather than whether they are 


Safe for use. 


In the past, UL wrote and published standards in a more or less closed forum. In 
the last 10 years or so, this process has become much more open. Today, more 
and more standards are being written by committee. Technical panels, consisting 
of government, industry, consumer, and UL representatives, are meeting together 
to develop and evolve the standards. This is a good thing. Collaboration and 


participation is encouraged at every level in the development of safer products. 


The first step in determining a standard is, then, assembling an expert panel who 
will then bring the knowledge needed to create the best set of requirements to 
achieve the goals of quality, sustainability and safety. A standard must be 
rigorous and practical — in that it has real world application, so manufacturers 
know they can practically produce a product that conforms to the standard. We 
have found that when we bring all stakeholders together, they are quite willing to 
put all their thoughts, opinions and ideas on the table and go through the pull and 
tug process of working toward the best solution. By achieving consensus, 
participants go away from the process saying ‘maybe | didn’t get everything | 
wanted, but | have to admit that we ended up with the best set of requirements’. 
And once a standard is published, all of us continuously look at real world 
application and new technologies that can change and improve the standard. 


Standards are living documents. 


Once a standard is accepted the next element in our safety model comes into play, 


Conformity Assessment. 


Conformity Assessment is the process of determining that a product 
conforms to a standard and that it continues to remain compliant. There are 


six functions involved in Conformity Assessment: 


The testing function involves the tasks that gather and record the data 
needed to make determinations of compliance. The tasks include, but are 
not limited to developing a test plan, conducting laboratory testing, 
recording test results, making physical measurements of product 
characteristics and reviewing markings and accompanying documents. 
Consideration must also be given to the need for accreditation of the 
laboratories conducting the testing. Laboratory accreditation is essential to 


ensure confidence in and consistency of test results. 


The determination of compliance function compares the information and data 
gathered during testing to the technical requirements to make a decision if 


that information and data demonstrates compliance with requirements. 


Conformity can be self-declared or assessed by 3 parties. The form of this 
is a suppliers declaration of conformity when done by the manufacturer and 
the certification function - a Mark - when conducted by an independent third 
party or government. This attestation normally takes the form of a 
supplier’s declaration of conformity document when the_ conformity 
assessment is conducted by the first party and a certification mark when 
issued by an independent third party. When the government is the 
conformity assessor the attestation can come in a variety of different forms 


such as formal letters from regulatory agencies and/or on product marks. 


Factory inspection is a pre-market mechanism that confirms compliance of 
on-going production. It usually involves physical inspection of products in 
production, verifying components, evaluation of production’ testing 
equipment/procedures and a review of the quality system its implementation 
and, often, sample testing. Results of factory inspection can be used to 


prompt/direct corrective actions or even to stop shipments when needed. 


Market Surveillance is a post-market function directed at validating the 
conformity of products that are available in the market. Tasks include testing 
and inspecting products obtained from the market to verify their compliance 
with technical requirements and the attestation of conformity made by the 
Supplier, independent third party or government following the_ initial 
determination of compliance. Results of market surveillance can be used to 
prompt corrective actions, improvements in the conformity assessment 


system and/or technical requirements when needed. 


Corrective action is the function that addresses tasks involved with 
addressing non-conforming products before they reach the market as well as 
misuse of the attestation of conformity. The tasks involved may include 
issuing public notices, conducting recalls and/or restricting the use of 


certification marks. 


UL has conducted conformity assessments for over 70,000 manufacturers in 
96 countries. In 2005 alone UL staff evaluated over 19,000 different types of 
products. We also conducted over 500,000 on-site factory follow-up visits to 
check on manufacturers’ continuing compliance with product certification 


requirements. 


Conforming to standards and conformity assessments require significant 
investments in time and money. For this reason, it is extremely important 


to protect those who play by the rules from those who don’t. 


Trademark counterfeiting is estimated at $500 billion annually or roughly 
5%-7% of global trade. Counterfeiting is a growing threat to every industry. 
| can tell you, UL is an organization that goes to great effort and expense to 


stop it. 


Our Anti-Counterfeiting Operations has two main objectives: first, to help 


protect the welfare and safety of consumers around the globe, and second, 


to protect the integrity of the UL Mark and all that it stands for. UL does not 
tolerate the counterfeiting of its Marks and takes swift and definite action 
against those engaged in this criminal activity. Consumers and our 
customers should expect their safety certifier to have a zero-tolerance policy 


when it comes to counterfeiting. 


UL has one of the most comprehensive intellectual property training 
programs in the world: We train personnel at 40-50 US Customs ports every 
year and we conduct law enforcement training seminars around the globe. 
Our aggressive program has trained over 2500 law and code enforcement 
officials since our program’s inception in 1995. On average, there are more 
than 100 US Customs seizures each year of products bearing counterfeit UL- 
marked products—with retail values estimated at over $12 million. Our 


training programs are working. 


We are taking the fight to the counterfeiters - working on the task forces of 
the FBI, Interpol, the World Customs Organization, and the International 
Trademark Association. We are involved in ongoing surveillance and raids - 
striking at the heart of the problem in China and South America. We are also 
partnering with the Royal Canadian Mounted Police combating counterfeits in 
their marketplace. And we’re working closely with our customers and 
industry trade associations to help them understand the scope of this 
problem and assist them in the development of their own anti-counterfeiting 


programs. 


Finally, yet equally important in our safety model, is the education and 
training of each and every entity in the marketplace, from manufacturers 


and their suppliers, to acceptance authorities, to end-users. 


UL University offers our customers over 1,500 courses providing safety 


standards training in areas such as Medical Devices, Medical Software, 


Industrial Control Panels, NEC, Plastics, Fire Safety, Hazardous Locations, 
Quality Training, OSHA, ITE and many more. We have a Hazard-Based 
Safety Engineering curriculum that helps engineers design safety into 
products by anticipating user behavior and related hazards. We teach our 


customers how to integrate quality and safety into their products by design. 


For acceptance authorities, we provide seminars that emphasize the 
importance of purchasing products that are certified as compliant with safety 
standards; what to look for when buying or receiving products covered by UL 


requirements; and how to detect and prevent counterfeiting of the UL Mark. 


And for end-users, the general public, we utilize mass media, we conduct in- 
store promotions; we go into schools and universities with consumer 
education and awareness programs and we support programs that teach 
safety; how to use these products safely; even, how to live safely, at home, 
at work and in public spaces. For example, we support the Center for 
Campus Fire Safety which teaches young people how to avoid dorm fires, 
how to escape them when they occur and how to use basic safety equipment 


like a fire extinguisher. 


This is our model for assuring safety in a competitive market. At UL, we are 
incredibly proud of our role in bringing safety to the marketplace and the 
countless lives that have been saved and injuries that have been prevented 
here at home and around the world. It is a model that has proven to be 
extremely effective over time. And it is a model that we believe can be 
used by defense and homeland security to help assure quality and 
sustainability in a competitive market. 


Recent events both at home and abroad have put immense pressure on the 
NDI community and have created an environment that can affect the 
performance and life-cycle costs of products. Namely: 


e ahigh demand for new products and services 


e arush to market to meet that demand 


e and the deployment of existing products in environments and for 
usage for which differ from their original design intent. 


The DoD, for the first time last year, soent more on maintenance and 
repair than on procuring new equipment. While certain performance 
issues may be unavoidable in these unpredictable times, it can be argued 
that with a more rigorous infrastructure in place, many instances of poor 


performance could be mitigated — if not immediately, then quickly. 


Now | want to say that this is not the first time UL has presented the inner 
workings of product and public safety to the defense and homeland security 
industry. On many occasions, Federal officials have come to our offices to 
learn about the regulatory infrastructure that fosters safety. We have heard 
many times, “This is what we need” and “How can we do that?” And we say, 
all you have to do is to determine that this is what you want. You are the 
manufacturers authorities having jurisdiction and, just as the insurers and 
building owners told the electrical manufacturers decided over a hundred 
years ago, you too can demand that the products and services your industry 
produces and uses meet a set of requirements, a standard, that can be 
tested and certified, which will assure quality and sustainability in a 


competitive market. 


Clearly the national defense industry is different from the _ civilian 
marketplace. And we recognize a very significant difference between 
defense and homeland security. That being said, however, there is an 
infrastructure in place, as | have just shown you, which can conform to the 
unique needs of defense and homeland security. An infrastructure that can 
helo the industry produce standards, test for compliance, prevent 
counterfeiting, and educate and train all entities involved in defense and 


homeland security. 


So let’s take a look at our safety model once again and discuss ways that it 


can be applied to the NDI community. First, the development of standards. 


The first thing NDI can do is review applicable and existing private sector 
standards and technical regulations that cover specific types of usage 
beyond typical commercial application. Where standards lackn requirements 


for such usage, new ones can be developed. 


Second, agree on and create conformity assessment and follow-up service 


models that suit your industry. 
Third, agree on and create a Marking system that suits your industry. 


In the safety model, standards often are developed through balanced, 
consensus-based, and transparent forums. This is a_ time-consuming 
process, but one that is right for the private sector— not necessarily for your 


industry. 


For reasons of both speed and security, standards can be developed in a 
closed forum. The companies and organizations here today represent world 
leaders. Ten of your people could sit down and in the course of a few 
months develop appropriate requirements that address usage, environment, 


life cycle and other objectives. 


In some cases, for security reasons, a standard may never be openly 
published. However, it is advisable, whenever possible, to make a standard 
aS open as possible — available to as many companies as_ possible. 
Competition leads to innovation and innovation ultimately drives down costs. 
Additionally, published standards send two very clear signals to the business 
community. One, that everybody will be producing to the same standard, 


and two, there will be a readily accepting market for their products. Every 


time you publish a standard you are assuring quality, sustainability and 


safety in that market. 


There is one more thing | would like to say about standards developed for 
Homeland Security. In addition to requirements that address functionality, 
product safety requirements should be part of the standard. Those who 
have the authority to purchase homeland security products — such as fire 
departments, airport authorities, and transportation authorities, at the local, 
state, and federal levels —need to know that the products they are using are 
not only effective, but also that they will not cause harm to the general 
public from their usage. In a recent poll, ninety-seven percent of electrical, 
building and fire AHJ’s, ninety-seven percent, said they can accept UL 
certified products without additional steps. When they see UL, and they 


know it meets their safety requirements. 


Conformity Assessment systems in the NDI community, like developing 
standards, may vary from product to product. When conformity assessment 
is desired we would see a two-track solution; using third parties like UL for 
Safety, Environment and Life Cycle Assessment and using DoD and DoE 
facilities for assessing functionality. Why do we recommend this approach? 
Simply, it is to apply the best resources to the task. For example to assess 
the functionality of products such as CBRNE detection devicesthere are only 
a few places in the US where you can test a biological detection device. As 
much as we are committed to safety | can say that neither UL nor anyone 
else in our industry is likely to get excited about testing with Anthrax and | 
suppose that you would also prefer that we not have such materials in our 
labs. Once functionality is established, an identical device should be tested 
for safety. Will the device work 24/7. Can it handle electrical surges? Will 


it cause shock or fire if it gets wet? Is the mounting device sufficient to 


protect those who walk under it? These are all issues that need to be 


considered when deploying these types of products in public spaces. 


In most instances, where product contracts will be awarded to multiple 
vendors, it is advisable to utilize 3 party conformity assessment, to verify 


consistency in the quality of all product coming off the line. 


3 party Conformity Assessment brings value to all members of the NDI 
community. For manufacturers, it enables International Market Access and 
provides continued assurance that products will be accepted by AHJs and 
other stakeholders. For AHJs, 3™ party Conformity Assessment provides a 
“level-playing field” of proven requirements instead of manufacturer 
assertions and unverifiable performance claims. And for Homeland Security 
and Defense entities, requiring 3 party Conformity Assessment ensures 
performance-based standards and related requirements are consistently met 
for products. And, as | have stated, this is not re-inventing the wheel. 
Ready-made 3” party Conformity Assessment processes are available today, 
into which DHS and DoD may plug in its broad set of needs and specify 


products that meet listing requirements. 


Now let me talk about counterfeiting and how it could affect your 
community. Today, in China, a sprinkler head is being manufactured with 
the UL mark on it that looks exactly like the sprinkler heads of legitimate 
Suppliers. They are being installed in buildings but they don’t work in 
extreme environments - like in a fire. Which is quite a scary thing. Ifa 
group of unseemly players takes the time to knock off a product that costs a 
dollar fifty, what’s to stop a counterfeiter from knocking off a five thousand 
dollar box that intended to protect Homeland Security. Especially in the 
Homeland Security marketplace, it is essential that an anti-counterfeiting 
program be put in place to protect the public and the legitimate players in 


the market. 


Education and training is the glue that holds our safety model together, that 
keeps everybody working on the same page. And it should play an 


important role in maintaining quality throughout your marketplace. 


As | mentioned at the beginning of my address, UL has a long history 
supporting defense and homeland security and we continue to be an active 
player in this arena. We believe this role can be expanded. We have an 
infrastructure in place — the resources, facilities, people and processes — 
that can help the NDI community reach its goal of sustaining quality in a 


competitive market. 


Now | realize a presentation such as this perhaps raises more questions than 
it does provide solutions. So | look forward to answering some of your 
questions, along with Dr. McQueary, after his presentation. But before | close, 
| just want to say, and | think | can speak for everyone here today, and say 
that nothing is more important than providing our war fighters and first 
responders with the highest quality equipment and services they need to 
accomplish their missions and which will protect them in the process. 
Nothing is more important than protecting citizens with the products and 
services we all need to keep us as safe and secure as possible here at home. 
Anything less is a disservice to those who put their lives at risk and to the 


American taxpayers who pay for the products and services that we provide. 


| will say it once again. The organizations here today represent world 
leaders. Your ingenuity, resourcefulness and dedication continue to produce 
technological advancements that provide an extraordinary service to our 


country. 


| truly believe that working together we can create, maintain, and sustain a 
culture of quality that continuously and predictably produces products and 


services that meet the needs of those who protect us. Thank you 
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Assuring Product Safety ina 
Competitive Market 


e Development of Standards 

¢ Conformity Assessment 
Factory Surveillance 
Protection from Counterfeiting 
Education and Training 
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Conformity Assessment 


¢ Testing 
Determining Compliance 
Issuing a Attestation of Conformity 
Factory Inspection 
Market Surveillance 
Corrective Action 
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Assuring Safety in a Competitive 
Market 

e Development of Standards 

« Conformity Assessment 

e Protection from Counterfeiting 


e Education and Training 
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Trademark Counterfeiting 
¢ 500 Billion Annually 
¢ 5%-7% of Global Trade 
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Anti-Counterfeiting Main Objectives 


e Protect Welfare and Safety of Consumers 


e Protect Integrity of the UL Mark 


UL Intellectual Property Training 


e Train personnel at 40-50 US Customs ports 
every year 


Trained over 2500 law and code 
enforcement officials 


Annual average of over 100 US Customs 
seizures 


e Retail values estimated at over $12 million 
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Applying Our Safety Model to the 
NDI Community 


e Development of Standards 
¢ Conformity Assessment 
e Protection from Counterfeiting 


e Education and Training 
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Synthesis Panel Points 
From NDIA SE Division’s 
DT&E Committee Co-Chairs 


Tom Wissink John Lohse 
Lockheed Martin Raytheon 
Fellow Engineering Fellow 


Date: March 15, 2007 


Some Opening Thought 


Bumper Stickers for Testers: 


Testers don't make the systems Warfighters use; 
Testers make those systems BETTER! 


Testing: You make it, We break it. 

A good Tester has the heart of a developer 
...1N a jar on their desk! 

Pertempto ergo sum 

| test, therefore | am 
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Key Observations & Comments 


Heard several times and agree that T&E (DT & 
OT) need to be involved earlier in SE process 


DT&E Cmte plans to work this item in 2007 

True for acquisition and implementation 
More Joint or Combined DT/OT 

There needs to be specific guidance 
Sustainment needs to be designed into systems 


Need guidance on what “Sustainment Testing” 
could look like in CT/DT/OT timeframes 


Agree with need “More/Better SEs and PMs” 
What about TEs (Test Engineers) 


March 15, 2007 DT&E Committee Co-Chairs @) 


Other Learnings, Comments & Thoughts 


What are the Industry incentives that would ensure 
desired Sustainability and Suitability? 


Sustainability (or is that “Material Availability”) KPP: 
Deliver higher TRL with less performance at lower 
ownership cost 


Logistics & Maintainability Demos performed in 

controlled environment (like DT) with parts available 
Needs to be more realistic 

Graduated Reliability during development 


There is a place for Standards — SOME should be 
updated and brought back into use 


Tailored for Rapid Deployment, Major Acquisition, etc 
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A Final Thought 


Many of these take Upfront Investment 
Fundamental change is needed 
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NEXT GENERATION EDS CT 
SYSTEMS 


George L Zarur, PhD 
Science Advisor 
S&T DHS 


Background & Current Systems 


Automated Explosives Detection Systems 
for Aviation prompted by Pan AM 103 


Design Genesis Medical CT 


An important Distinction: 

Aviation EDS CT has a significantly higher 
duty cycle. 400 bags per hour with newer 
systems 800 to 1000 bags per hour. 


e Excellent Reference material 


e National Academies Press 1998 ( 
Configuration Management and 
Performance Verification of Explosives- 
Detection Systems ) 


e Side view as a standalone System 


¢ Optimal Configuration as an inline System 
with the Airport Baggage Handling System 


= 


e¢ Limited Deployment in pre 911 plans 


¢ Deployment by Airlines for limited 
screening 


¢ Post 911 with passage Of ATSA Nov 19 
2001, 100 percent screening of checked 


luggage 


e Design dictated by the need for very high 
detection rates and 3 to 400 bags per hour 
throughput, speed of the rotating gantry, 
and the belt influenced the resolution and 
the size of the volumetric voxel under 
inspection 


¢ Basic technology driven by Xray Source 


e Detection based on density discrimination 
ina similar fashion to cancer detection. 
Unlike Medical CT, shape is never 
considered in detection. 

¢ CT EDS systems are very effective at 
Detection and approaching optimal 
conditions. 


RDT&E Efforts 


e The testing and Certification Efforts at the 
Transportation Laboratory has developed 
a mature and effective process, that has 
resulted in a greater improvement in 
reliability and service levels with improved 
detection levels. 


Next Gen EDS Drivers 


Overall Guiding Principle is based on Best 
Value and Affordability 


There is a significant data collected over 
the past few years on O&M costs 


Reduce Capital Acquisition Costs 
Installation Costs 
Field maintenance Costs 


e False Alarm Rate Reduction has 
Significant Impact on operational costs, all 
alarms must be resolved and requires 
some sort of manual inspection, adding to 
labor and other system costs. 


e A major distinction between Medical and 
Security systems can be traced to two 
factors. 


e Passenger bags vary significantly both in 
size and content with huge clutter and 
many layers of material with varying 
densities and textures. 


e¢ Threats do not have unique density values 
and share the domain space with many 
harmless and innocuous substances 


Next Generation Design goals 


e Promotion of standards at various 
interfaces to reduce reliance on 
proprietary designs. 


e Attempt to segregate the hardware from 
the Detection Algorithm, (not unlike 
implementation of the DNDO Advanced 
Spectroscopic Portal), Implement Image 
Standards, similar to medical CT 


¢ Consideration of Non Rotating gantry 


designs with novel X-ray sources. Instead 
of a single source rotating at high speed, 
multiple sources, preliminary designs 
upward of 500 to 900 sources distributed 
in a circular fashion. 


Investigate the benefit of increased 


resolution through source characteristics 
and detector arrays. 


e Investigate novel x-ray sources such as 
Nanotube technology with longer lifetime 
and lighter source assemblies and 
facilitate the use of dual energy for threat 
discrimination. Commercial units are likely 
end of CY0O7. Promise of significant 
lifetime and reduced costs, reduction in 
AC requirements. 


e Investigate inclusion of add on 
Technologies which derive a chemical 
composition discrimanants, such as 
Coherent X-ray, where collected spectra 
relate to the individual chemical 
composition of a threat. Promising 
preliminary systems are in service. 


Why Emphasis on False Alarm 
Reduction 


e Although algorithm improvements have 


reduced the false alarm rates for current 
CT technology, rates exact an operational 
cost that can and should be reduced. 


Novel threats, such as Home made 
explosives and liquid explosives, often 
with a wide range of density and physical 
characterisitics, can lead to a significant 
increase in False Alarm rates 


2007 versus 2001 


e TSA has a mature and capable operation 
to collect and analyze operational data for 
a comprehensive life cycle cost 
management not available in late 2002 as 
CT EDS systems were deployed. 


¢ Technology Advances in both material and 
Software with a larger pool of prospective 
developers with interest to compete for the 
Next Gen System Deployment. 


Presentation by Mr. John Stoddart 
23” Annual National Test & Evaluation Conference 
March 14, 2007 


Sustainability KPP: Industry Weighs in on 
Bidding for It and Building for It 


GOOD MORNING AND THANK YOU FOR THAT KIND INTRODUCTION 


| APPRECIATE THE INVITATION TO SPEAK TO YOU ABOUT THE 
SUSTAINABILITY KPP. MY MESSAGE IS PRETTY SIMPLE: INDUSTRY HAS 
THE WHEREWITHALL TO DESIGN AND BUILD AFFORDABLE SYSTEMS 
THAT CAN DRAMATICALLY IMPROVE SUSTAINABILITY AND 


REVOLUTIONIZE LOGISTICS. 


WE, IN INDUSTRY, PROVIDE FOR SUPPORTABILITY OF THE SYSTEMS WE 
PRODUCE THROUGH A WIDE RANGE OF MEASURES, SUCH AS 
THOUGHTFUL DESIGNS, MANUALS, OPERATOR AND MAINTENANCE 
TRAINING, PROVISIONING FOR SPARE PARTS, AND CONTRACTOR 


LOGISTICS SUPPORT. 


REGARDLESS OF HOW THE GOVERNMENT ASKS FOR SUSTAINABILITY, 
IT IS ANECESSARY ELEMENT OF ANY PROGRAM. BUT THE PROCESS OF 
ASSURING THAT SYSTEMS ARE SUSTAINABLE HAS NOT WORKED AS 


WELL AS IT COULD. 


SYSTEMS THAT HAVE RELIABILITY PROBLEMS HAVE OBVIOUS IMPACTS 
ON WHETHER OUR SERVICE MEN AND WOMEN CAN MEET THEIR 
MISSION REQUIREMENTS. BEYOND MISSION SUCCESS - HIGH 
RELIABILITY MEANS LESS INVESTED IN REPAIR PARTS, THE SUPPLY 
CHAIN THAT BRINGS THEM AND ULTIMATELY IN THE COST OF 


OWNERSHIP. 


MAINTAINABILTIY PROBLEMS HAVE A MORE SUBTLE, BUT NONE-THE- 
LESS EXPENSIVE, IMPACT ON THE SERVICE AND ITS SUPPORT SYSTEM 
IN THAT REQUIRED MAINTENANCE MANHOURS CORRELATE DIRECTLY 
TO THE REQUIRMENT FOR MAINTENANCE PERSONNEL AND TECHNICAL 
SUPPORT. THE SIMPLER AND EASIER A SYSTEM IS TO MAINTAIN, THE 


FEWER MAINTAINERS THAT ARE REQUIRED. 


DESPITE THE IMPACT ON MISSION PERFORMANCE AND OWNERSHIP 
COSTS, THE “ILITIES” OFTEN PLAY SECOND FIDDLE TO ATTRIBUTES 
DEEMED TO BE OF HIGHER PRIORITY BY THE GOVERNMENT, SUCH AS 


MOBILITY, LETHALITY, AND PROTECTION. 


THE “TRADES” GENERALLY MADE AS REQUIREMENTS (INCLUDING KEY 


PERFORMANCE PARAMATERS, OR KPPS) ARE TRANSLATED INTO 


PERFORMANCE SPECIFICATIONS, WHICH IN TURN, ARE TRANSLATED 


INTO CONTRACTUAL DELIVERABLES. 


WHY TRADE THE “ILITIES’? USUALLY IT IS BECAUSE OF THE PRECEIVED 
IMPACT ON COST AND THE ABILITY TO ACHIEVE TESTABLE 


VERIFICATION THAT THESE REQUIRMENTS HAVE BEEN MET. 


| BELIEVE THE DEFENSE INDUSTRY THAT | AM A PROUD MEMBER OF 
CAN MEET SUSTAINABILITY REQUIREMEMTS, BUT WE SHOULDN'T 
EXPECT JUST TO BE ABLE TO SAY “DEAR SANTA, | WANT IT ALL!” 
ACHIEVING SUSTAINABILITY PARAMETERS IS POSSIBLE, BUT IT WON'T 
BE FREE. EVEN WITH ALL THE TECHNOLOGICAL ADVANCEMENTS 
WE’VE MADE, THESE ARE STILL MACHINES — AND OUR MACHINES CAN 
BE COMPLEX, MAKING THE OPPORTUNITY TO HAVE FAILURES EQUALLY 


AS COMPLEX. 


BUT FINDING WAYS TO DESIGN SUSTAINABILITY INTO SYSTEMS, 
FINDING WAYS TO MEASURE SUSTAINABILITY EFFECTIVELY AND FAIRLY 
AND REDUCING THE WARFIGHTER’S BURDEN IS WHAT WE ALL SHOULD 


BE ABOUT. 


| OFFER TO YOU THAT INDUSTRY CAN DESIGN SYSTEMS THAT CAN BE 
SUSTAINED WITH LOWER BURDEN ON THE WARFIGHTER. CONSIDER 


THE FOLLOWING. 


AT A RECENT CONFERENCE SIMILAR TO THIS ONE, A SPEAKER POSED A 
QUESTION TO THE ATTENDEES ABOUT MAKING PREPARATIONS FOR A 
TRIP IN THEIR CARS. IN THE PROCESS OF GETTING READY, HE ASKED 
IF THE ATTENDEES WOULD PACK A TOOL SET AND CARRY A SUPPLY OF 


SPARE PARTS. 


FOR THAT MATTER, WOULD THEY, OR YOU, HAVE ANXIETY ABOUT 
WHETHER OR NOT THE CAR WOULD START WHEN THE KEY IS TURNED 
IN THE IGNITION, OR IF IT WOULD BREAK DOWN UNEXPECTEDLY? | 
THINK YOU WOULD AGREE THAT MOST OF US DO NOT GIVE SUCH 
MATTERS A SECOND THOUGHT BECAUSE COMMERCIAL AUTOMOBILES - 
AND TRUCKS FOR THAT MATTER - ARE QUITE RELIABLE. COMMERCIAL 
VEHICLES ARE RELIABLE BECAUSE OF THE COMPETITIVE MARKET 
PLACE AND BECAUSE WE, AS CONSUMERS, HAVE COME TO EXPECT 


AND DEMAND IT. 


IN THE 1950s AND EARLY 1960s, AS CARS FROM JAPAN BEGAN TO GAIN 
A FOOTHOLD IN THE U.S. MARKET, THEY WERE NOT NOTICEABLY 


BETTER THAN DOMESTIC AUTOMOBILES. IN FACT, TO MANY THEY 


SEEMED CHEAP, SMALL, AND RATHER ORDINARY COMPARED TO U.S. 


MODELS OF THAT ERA. 


OVER TIME WE ALL WATCHED AS THE JAPANESE AUTO MAKERS MADE 
QUALITY AND RELIABILITY A PRIORITY AND WE HAVE SEEN THE IMPACT 
OF THEIR PHILOSOPHY IN REVERSALS OF MARKET POSITION AND EVEN 
THREATS TO LONG TERM SURVIVAL OF SOME FAMOUS U.S. BRANDS. 
BECAUSE OF COMPETITION, QUALITY OF U.S. CARS HAS GOTTEN 


BETTER, EVEN IF THE PERCEPTION HAS NOT CAUGHT UP. 


WHAT DOES THIS HAVE TO DO WITH MILITARY VEHICLES AND MILITARY 
EQUIPMENT IN GENERAL? SOME IN INDUSTRY AND THE GOVERNMENT 
BOAST OF OPERATIONAL READINESS RATES FOR MILITARY SYSTEMS IN 
THE LOW 90s. IN SOME CASES, SYSTEMS’ READINESS LEVELS MAY 
SIMPLY BE A FUNCTION OF AGE, OR HOW HARD IT IS BEING USED, OR 
EXTRAORDINARY MEASURES TO KEEP THEM UP. WELL, A 90-SOME 
PER CENT READINESS RATING MAY BE ACCEPTABLE - EVEN A CAUSE 
FOR EUPHORIA - FOR MILITARY EQUIPMENT IN A THEATER OF 
OPERATIONS; HOWEVER, YOU WOULD NOT ACCEPT 90 PER CENT FOR 
YOUR LEXUS, CAMRY, OR FORD. YOU FULLY EXPECT 99.999 PER CENT! 
THAT IS WHY YOU DON’T BOTHER TO PACK THE TOOL CHEST AND 
SPARE PARTS, OR WORRY ABOUT NOT STARTING OR BREAKING DOWN 


AT THE LEAST OPPORTUNE MOMENT. 


FOR US, BREAKING DOWN WOULD BE AN INCONVENIENCE THAT MIGHT 
SHAPE YOUR NEXT CAR PURCHASE DECISION. BUT BREAKING DOWN 
FOR PEOPLE IN COMBAT IS THE DIFFERENCE BETWEEN LIFE AND 
DEATH. SYSTEMS THAT FAIL TOO FREQUENTLY ARE HARD TO SUSTAIN, 
DRAINING COMBAT POWER AND CAUSING A RIPPLE EFFECT 


THROUGHOUT THE FORCE. 


WHAT ELSE IS DIFFERENT BETWEEN YOUR CAR AND MILITARY 
EQUIPMENT? YOU DON’T CHECK UNDER THE HOOD BEFORE YOU 
OPERATE YOUR CAR. YOU DON’T GO THROUGH A PREVENTATIVE 
MAINTENANCE CHECKLIST. NO, ASSUMING YOU HAVE KEPT PACE WITH 
TECHNOLOGY, INSTEAD A COMPUTER CHECKS THE VITAL SYSTEMS IN 
YOUR CAR AND TELLS YOU IF THE BRAKES PADS ARE WORN, LIGHTS 
ARE INOPERABLE, FLUIDS ARE LOW, OR IF THERE IS A PROBLEM WITH 
THE ENGINE. IF THERE IS A PROBLEM, THE COMPUTER TELLS THE 
MECHANIC WHAT IS WRONG FOR MOST OF THE VEHICLE’S CRITICAL 
SYSTEMS. SOME CARS CAN COMMUNICATE WITH THE MANUFACTURER 
AND SEND YOU A REMINDER OF AN UPCOMING SERVICE OR WITH AN 


EMERGENCY SERVICE IF YOU ARE IN AN ACCIDENT OR LOST. 


THE MILITARY HAS BEEN TRYING TO HEAD IN THE SAME DIRECTION FOR 


SOME TIME NOW. THE REASON THERE HASN'T BEEN MORE PROGRESS 


IS NOT BECAUSE INDUSTRY CANNOT DO IT, OR BECAUSE MILITARY 
SYSTEMS ARE UNIQUE, OR BECAUSE SUCH FEATURES ARE 
UNAFFORDABLE BELLS AND WHISTLES. WHEN JAPANESE CAR MAKERS 
WERE BEGINNING TO BUILD THEIR HIGHLY RELIABLE REPUTATIONS AS 
COMPARED TO U.S. (AND EVEN EUROPEAN) BRANDS, WAS THERE A BIG 
DIFFERENCE IN THE SELLING PRICES OF MORE RELIABLE CARS AS 
OPPOSED TO LESS RELIABLE ONES? NO. THE JAPANESE WERE ABLE 
TO DESIGN FOR GREATER RELIABILITY AND SUSTAINABILITY WHILE 
REMAINING PRICE COMPETITIVE BECAUSE THEY HAD TO IN ORDER TO 


MEET THE DEMANDS OF THEIR CUSTOMERS AND GAIN AN ADVANTAGE. 


BUT PRICE COMPETITIVENESS IS WHERE MOST OF US IN INDUSTRY LIVE 
AND IF OUR CUSTOMER IS WILLING TO LET SUSTAINABILTIY BE A “NICE 
TO HAVE” RATHER THAN A “MUST HAVE,” THEN NONE OF US CAN 
AFFORD TO TAKE THE RISK THAT WE’LL HAVE OVER BUILT THE 
REQUIREMENT AND PRICED OURSELVES OUT OF THE COMPETITIVE 


RANGE. 


THERE ARE BRIGHT SPOTS IN THE SUSTAINABILITY FUTURE — FOR 
EXAMPLE, IN THEIR LOGISTICS SUPPORT VEHICLE REPLACEMENT, OR 
LVSR, THE MARINES WANTED TO SIMPLIFY LOGISTICS SUPPORT SO 
THEY IMPOSED A REQUIREMENT FOR A SINGLE LUBRICANT FOR THE 


ENGINE AND TRANSMISSION AND A SINGLE RESEVOIR. 


WE AT OSHKOSH WERE SUCCESSFUL IN PUTTING THAT FEATURE IN 


THE DESIGN AS PART OF A BROAD PLAN TO IMPROVE SUSTAINABILITY. 


HYBRID ELECTRICS AND ONBOARD POWER GENERATION CAPABILTIES 
ARE GENERATING MOMENTUM AND THESE CAPABILITIES HAVE THE 
POTENTIAL TO MAKE VEHICLES SIMPLER AND MORE SUSTAINABLE. 
HYBRID ELECTRIC TECHNOLOGY CAN REDUCE FUEL CONSUMPTION, 
AND DUE TO ITS ABILITY TO PRODUCE EXPORTABLE MILITARY GRADE 
ELECTRICAL POWER, IT ALSO COULD REDUCE THE REQUIEMENT FOR 
TRAILERS AND POWER GENERATORS. THIS, IN TURN, MEANS FEWER 


SYSTEMS WOULD HAVE TO BE SUSTAINED. 


THERE IS ANOTHER HUGE OPPORTUNITY ON THE TABLE RIGHT NOW IN 
THE JOINT LIGHT TACHTICAL VEHICLE. AS THE USER COMMUNITIES 
STRUGGLE TO BALANCE JLTV’S THREE MOST IMPORTANT ATTRIBUTES: 
PERFORMANCE, PROTECTION, AND PAYLOAD, WHERE WILL RELIABLITY 
AND MAINTAINABILITY FALL AMONG PRIORITIES FOR A FLEET THAT WILL 
BE POTENTIALLY THE HIGHEST DENSITY MOBILITY SYSTEM IN EITHER 
THE ARMY OR THE MARINE CORPS? | CONTEND THAT IMPROVED 
SUSTAINABILITY CAN BE PROVIDED WITHOUT HAVING TO TRADE 


MISSION CAPABILITIES. 


FROM MY PERSPECTIVE, WE CAN DO BETTER. WE CAN DO BETTER AND 
IT DOES NOT NECESSARILY MEAN THAT COSTS WILL SUDDENLY MAKE 
SYSTEMS UNAFFORDABLE. WE CAN DESIGN VEHICLES THAT ARE 
MODULAR, SO COMPONENTS CAN BE SWAPPED OUT IF THEY BECOME 
INOPERABLE, BATTLE DAMAGED, OR IF NEWER TECHNOLOGY COMES 
ALONG. THEY CAN BE INHERENTLY MORE RELIABLE AND MORE EASILY 
REPAIRABLE BY ASSURING EASIER ACCESS TO ALL ASSEMBLIES, 
LEADING TO IMPROVEMENTS IN SUSTAINABILITY. THEY CAN TELL THE 
OPERATOR WHEN SOMETHING HAS FAILED (OR IS ABOUT TO FAIL), THE 
STATUS AND LOCATION OF THE SYSTEM, AND MORE. IN OTHER 
WORDS, THEY CAN BECOME SOMETHING MORE LIKE YOUR 


TRUSTWORTHY CAR. 


WE JUST NEED OUR CUSTOMER TO REQUIRE IT FROM US AND WE NEED 
TO KNOW HOW WE WILL BE FAIRLY EVALUATED. THEN WE CAN DESIGN 
EQUIPMENT THAT IS DESIGNED ACCORDINGLY FROM THE GROUND UP. 


IF WE CAN DO THIS, THE IMPLICATIONS ARE HUGE. 


CONSIDER THAT LOGISTICS IS LARGEST PART OF THE FORCE ... SOME 
70% IN THE CASE OF THE ARMY. THE STRUCTURE DEVOTED TO 
LOGISTICS IS DIRECTLY RELATED TO THE AMOUNT OF FUEL, WATER, 
AMMUNITION, SPARE PARTS AND OTHER SUPPORT ITEMS NEEDED. 


AND, ALL OF THAT REQUIRES TRANSPORTATION OPERATORS THAT 


NEED TO BE TRAINED, EQUIPPED, SUSTAINED, PROTECTED, AND 


PROVIDED HEALTH CARE, AND SO ON. 


FOR EVERY REDUCTION IN DEMAND THAT WE CAN DESIGN INTO THE 
POINTY END OF THE SPEAR, WE CAN DRAMATICALLY REDUCE THE 


LOGISTICS BURDEN AND MAKE THE FORCE MORE SUSTAINABLE. 


AT THE END OF THE DAY WE WILL BID, BUILD AND DELIVER WHAT THE 
GOVERNMENT ASKS OF US. SUSTAINABILITY IS ACHIEVABLE — AND IT 
CAN BE A KPP, BUT A GREAT DEAL OF THOUGHT WILL HAVE TO GO 
INTO THE MEASURES OF SUCCESS AND THE PARAMETERS FOR 
TESTING IF SUSTAINABILITY IS NO LONGER “TRADE SPACE” AND MUST 


BE MET AS A CRITERION TO GO FORWARD WITH A PROGRAM. 


AND MOST OF ALL — EVEN THOUGH SUSTAINABILITY SHOULD BE 
AFFORDABLE — WE ARE GOING TO HAVE TO DO A BIT MORE THAN SAY 


“DEAR SANTA | WANT IT ALL.” 


THANK YOU FOR YOUR ATTENTION AND | LOOK FORWARD TO YOUR 


QUESTIONS AND A LIVELY DISCUSSION! 


